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Overview 


In 2004, Lumina Foundation for Education launched an innovative national reform initiative, 
Achieving the Dream (ATD), with the purpose of increasing the persistence and completion 
rates of community college students, especially low-income students and students of color, 
through evidence-based institutional change. Originally consisting of 26 partner colleges in five 
states, today more than 150 colleges in 30 states and the District of Columbia participate in 
ATD, which in 2010 became a national nonprofit organization. 

This report examines six community and technical colleges in Washington State that joined the 
initiative in 2006 (“Round 3”) and were provided with grant funding from College Spark 
Washington and the services of a leadership coach and data coach to facilitate their ATD 
efforts. The report (1) describes the progress each college made in implementing ATD’s 
“culture of evidence” principles for institutional improvement; (2) examines strategies 
implemented by the colleges to improve student outcomes, comparing them to interventions 
developed by the 26 “Round 1” colleges that joined ATD in 2004; and (3) charts trends in 
student outcomes in the period before and after the Washington colleges joined the initiative. 

Key findings for the six Washington ATD colleges are: 

• Progress toward building a culture of evidence. All but one of the 

colleges made at least some progress. Two colleges made substantial 
progress, moving from a low level of implementation to a high level over 
the five years of their ATD involvement. Specifically, they strengthened 
student success goals and strategies, increased their institutional research 
capacities, created institutional effectiveness committees, and strengthened 
their program review processes. 

• Development of student success strategies. Several of the colleges — 

those further along in implementing the ATD culture of evidence 
principles — made significant systemic changes to programs and services. 
Compared with the Round 1 ATD colleges, the Washington colleges were 
more likely to have implemented changes in instruction as opposed to 
student support services and were more successful in operating 
improvement strategies at scale. 

• Student outcome trends after ATD implementation. The average student 

outcomes across the six colleges appear largely unchanged, as do the racial 
and economic achievement gaps. The colleges that succeeded in 
implementing improvement strategies at scale did so only later in the 



period under study. It may be too early to see their impact. On the other 
hand, most of the reforms implemented by the Washington colleges were 
at the “front end,” in that they involved changes to orientation and intake 
advising as well as to developmental instruction. Improving student 
completion rates may also require systemic changes to the design of 
academic programs to ensure that they are well structured and aligned with 
the requirements for success in further education and, for career-technical 
programs, success in the labor market. 

The report also examines lessons from the experiences of the Washington ATD colleges for 
other colleges seeking to improve student outcomes. 

This study was conducted during a period of both rapid enrollment growth and sharp cuts in 
state funding. Making systemic changes in practice in ways that the Washington ATD colleges 
have begun to do and in the other ways recommended in this report requires resources — 
resources for new and strengthened information systems, for professional development and 
training, and for coordination. The Washington colleges have shown that even during a period 
of dramatic cuts in funding, they have been willing to make major changes in practice. The fact 
that they have reallocated scarce resources toward practices that have the potential to be more 
effective illustrates their commitment to greater student success. 
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Preface 


Open-access, low-cost community colleges are increasingly understood as a critical 
resource for millions of adults who might otherwise be unable to access higher education. 
Community college degrees and certificates are instrumental for many low-income students 
seeking jobs that pay a living wage. Yet it is also true that the majority of students who enroll in 
a community college do not graduate or earn a credential; indeed, many students drop out even 
before completing developmental coursework that is designed to prepare them for the academic 
challenges of college -level courses. 

Nearly a decade ago, Lumina Foundation for Education launched a bold initiative — 
Achieving the Dream — to help more community college students, particularly low-income 
students and students of color, stay in college and earn a certificate or degree. The initiative calls 
upon colleges to use data on their own students to evaluate institutional programs and services, 
thus providing a way for faculty and administrators to engage in thoughtful self-assessment, 
reflection, and decision making that leads to the implementation and evaluation of strategies 
aimed at increased student success. Achieving the Dream has grown dramatically since its 
founding in 2004 — more than 150 colleges in 30 states now participate in the initiative. 

Since Achieving the Dream began, the Community College Research Center at 
Teachers College, Columbia University, and MDRC have been examining the ways in which 
participation in the initiative has affected community colleges and their students. One analysis 
(Rutschow et al., 2011) covered the 26 institutions that joined the initiative initially in 2004. 
Another study (Jenkins, Ellwein, Wachen, Kerrigan, & Cho, 2009) reviewed the early 
Achieving the Dream experiences of 13 colleges — seven in Pennsylvania and six in 
Washington State — that joined the initiative in 2006. 

The study reported on here evaluates the changes to date in the six Washington 
colleges. It compares their progress after five years of participation in Achieving the Dream 
with, first, a baseline examination of their progress (Jenkins et al., 2009), and, second, with 
findings on the progress of the 26 original Achieving the Dream colleges (Rutschow et al., 
2011). The study shows that most of the Washington colleges made great progress in building a 
culture and an infrastructure for evidence-based improvement. Several of the Washington 
colleges have made significant changes to their programs and services, especially in intake 
advising and developmental education. Colleges that succeeded in implemented innovations at 
scale did so rather late in the study period, so it may be too early to see their effects. 

One clear implication of the study is that no single intervention that a college 
undertakes will likely lead to a large increase in college-wide persistence or completion rates. 
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Rather, the redesign of major programs and functions throughout a college must be considered 
in order to achieve substantial gains in student success. 


Thomas Bailey 

Director, Community College Research Center 
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Executive Summary 


This report presents findings from an evaluation of the work on Achieving the Dream 
(ATD) of the first six Washington State community and technical colleges to participate in the 
initiative. ATD is a national community college reform movement dedicated to increasing the 
number of students, particularly low-income students and students of color, who earn 
postsecondary credentials with marketplace value. Currently over 150 institutions in 30 states 
and the District of Columbia take part in ATD. The six Washington colleges examined in this 
report joined the initiative in 2006. These colleges, which in the body of the report are not 
identified by name in order to optimize learning from their experiences, are Big Bend 
Community College, Highline Community College, Renton Technical College, Seattle Central 
Community College, Tacoma Community College, and Yakima Valley Community College. 

This report is part of a larger evaluation of ATD being conducted by MDRC and the 
Community College Research Center (CCRC) with funding from Lumina Foundation for 
Education. The purpose of the Washington State study, also underwritten in part by College 
Spark Washington, was to examine the progress made by the Washington ATD colleges in 
implementing the ATD “culture of evidence” principles for institutional improvement and in 
improving student outcomes. This report on the Washington ATD colleges examines the 
following research questions: 

• Have the colleges made progress in building a culture of evidence by 
implementing the ATD principles of institutional improvement? 

• What student success strategies did the colleges implement under ATD, 
and which of these are being implemented at scale? 

• Has involvement in ATD changed the way that faculty, staff, and 
administrators use data? If so, in what ways? 

• What have been the trends in student achievement among the Washington 
ATD colleges? Is there evidence that student progression and outcomes 
improved during the period after the Washington State colleges joined 
ATD? 

Lumina Foundation and other ATD funders are especially interested to see if the 
Washington State colleges, which joined the initiative in the third round when it was more fully 
developed, progressed faster than colleges that joined in the first round. Hence, another focus of 
this report is to compare the progress of the Washington colleges over the first five years of 
their involvement in the initiative with that of the “Round 1” ATD colleges. 
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We collected data for this evaluation through a review of the Achieving the Dream 
internal reports, field research at all six colleges, a survey of data use by faculty and 
administrators, and student data shared by the colleges. The study was conducted in two waves: 
a “baseline” examination of the initial work and early progress of the Washington ATD colleges 
conducted in 2008, and follow-on research conducted in 2011, the fifth and final year of the 
colleges' initial implementation work on ATD. To assess the progress made by the Washington 
colleges, this report compares the findings from the second wave of research in Washington 
with that of the first wave, which were examined in a 2009 report by CCRC and MDRC 
(Jenkins, Ellwein, Wachen, Kerrigan, & Cho, 2009). It also compares the progress of the 
Washington colleges after their first five years in ATD with that of the first round of colleges to 
join the initiative, which was explored in a 201 1 MDRC-CCRC report (Rutschow et al., 201 1). 

It is important to point out that this study was conducted during a period of both rapid 
enrollment growth and sharp cuts in state funding. Thus, the Washington ATD colleges were 
faced with the challenge of making systemic changes to practice in a time of scarce resources. 


Progress Building a Culture of Evidence 

Table ES.l shows the progress made by the Washington ATD colleges in implementing 
practices that reflect the four Achieving the Dream “culture of evidence” principles: (1) 
leadership committed to improving student outcomes and equity, (2) use of evidence to improve 
student success, (3) broad engagement of stakeholders in the improvement process, and (4) 
continuous improvement. The ratings are based on a protocol and rubric the research team used 
to evaluate how much each college had implemented the ATD principles. For each dimension, 
the college received a rating from 1 (little or none) to 5 (high level/a lot). 

All but one of the colleges made at least some progress toward building a culture 
of evidence according to the ATD principles. Two colleges made substantial progress, 
moving from a low level of implementation to a high level over the first five years of 
their involvement with the initiative. 
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Achieving the Dream Colleges in Washington State 


Table ES.l 


Level of Implementation of ATD Culture of Evidence Principles at the Washington 
Colleges: A Comparison of Spring 2008 and Spring 2011 


College 

Spring 2008 

Spring 2011 

Increase in Levels 

WA-A 

High 

Very high 

+ 1 

WA-B 

Medium 

High 

+ 1 

WA-C 

Low 

High 

+2 

WA-D 

Low 

High 

+2 

WA-E 

Very low 

Low 

+ 1 

WA-F 

Very low 

Very low 

0 


NOTE: In this report, we do not name colleges but rather use aliases to optimize learning from their 
experiences. 


While it is difficult to compare the six Washington ATD colleges with the 26 Round 1 
ATD colleges, it does seem that the Washington colleges, as a group, went at least as far in 
building a culture of evidence as did the Round 1 colleges. Five years after they joined ATD, 
we rated four of the six Washington colleges as having implemented the ATD principles at a 
high level. This means that the college leadership had made strong commitment to improving 
student outcomes (not just enrollments), that the colleges were actively engaging faculty and 
staff and using student records and other data to assess the effectiveness of student services, and 
that they strengthened strategic planning and other infrastructure for continuous improvement. 
In comparison, only 1 1 of the 26 Round 1 colleges (or 42 percent) implemented most of the 
practices associated with a strong culture of evidence by the end of their five-year ATD 
implementation period. 

Bringing about changes in organizational practice is not easy. All of the Washington 
colleges faced barriers to implementing practices that reflect the ATD principles. At the college 
that made the least progress in building a culture of evidence, turnover among the college’s top 
leadership made it difficult to sustain a focus on student success. All six colleges initially had 
trouble accessing the data needed to conduct the longitudinal cohort tracking recommended by 
ATD because all of the colleges in the state were dependent on an antiquated statewide “legacy” 
student information system. Three of the colleges built data warehouses to facilitate longitudinal 
analysis of data, although the others continued to go through the cumbersome process of 
downloading data from the state system each time they conducted new analyses. 

It is also true that bringing about systemic change requires resources. Although the 
Washington ATD colleges received some funding to support ATD activities (about $100,000 
per year during the implementation year), the period under study was one in which per-student 
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revenues declined due to a sharp increase in student enrollments and sharp cuts in state funding 
in the wake of the Great Recession. One area essential to change efforts in which some colleges 
were forced to cut was professional development for faculty and staff. For example, one college 
eliminated a full-time faculty development staff person whom faculty interviewees praised for 
helping to support efforts to improve instruction. The alternative, according to the college’s 
administrators, was to reduce resources for instruction. 

Despite these challenges, all of the colleges made at least some changes to 
organizational practice as a result of their participation in Achieving the Dream. Table ES.2 
summarizes these changes. 


Achieving the Dream Colleges in Washington State 
Table ES.2 

Major Changes in Organizational Practice Among the Washington Colleges 
During the First Five Years of ATD 


Changes in Organizational Practice Number of Colleges 

Strengthened goals/measures for student success 6 

Increased IR staff and capacity 4 

Strengthened student success strategy development process 4 

Built data warehouse and added programmer 3 

Created institutional effectiveness committees 3 

Strengthened program review process 2 


All six colleges developed goals for student success using measures that were similar to 
and in some cases adapted from the ATD performance measures. One college went a step 
further by asking each department to establish student success goals and measures tied to 
institutional goals. Another strengthened its program review process, with performance 
measures defined in collaboration with faculty. 

Three of the colleges added institutional research (IR) staff who built data warehouses, 
improved procedures for requesting information, provided data support to student success 
committees and others, and trained faculty and staff in the use of data. The report includes a 
case study of one of these colleges that had struggled for years to produce data that college 
faculty and other stakeholders could trust. The college was successful in building a home-grown 
institutional effectiveness office, choosing a well-regarded faculty member to head the new 
office, which sent the message to the faculty that research would focus on instruction. In these 
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three colleges and one other, the efforts of the IR staff to increase access to data and research 
was enhanced by “data teams” comprised of faculty, staff, and administrators who helped to 
“democratize” the use of data at the college, promoting data use among their colleagues and 
also providing guidance on how to do so. 

The three colleges that added IR staff also created standing bodies that were responsible 
for overseeing efforts to improve student outcomes and institutional effectiveness. Such a 
committee already existed at another one of the colleges prior to its joining ATD. In some cases 
the ATD data teams were integrated into these committees; in others, the data teams were 
separate. In all cases, IR staff worked closely with committee members to help ensure that the 
decisions they made were based on evidence. All four colleges with such committees also 
created or strengthened processes for identifying opportunities to improve student success and 
to develop strategies for doing so. 

As part of its strategic planning and improvement process, one college worked with 
each department to set measurable goals for improving student learning and success. Another 
college formalized and strengthened its program review process to include a focus on student 
retention and completion, not just enrollment. The college used this process to decide which 
programs to eliminate in response to sharp cuts in state funding that occurred during this period. 
The administration worked with the faculty to develop the metrics to be used. Faculty and 
department chairs that were interviewed said that the administration was trying to be more 
transparent and fair in its decisions. The college’ s strengthened IR capacity and program review 
process and measures helped to make this possible. 


Student Success Strategies 

A spring 2011 report by MDRC and CCRC on the progress of the 26 Round 1 ATD 
colleges yielded three main findings regarding the implementation of strategies for improving 
student success (Rutschow et al., 2011). First, only one-fourth of all direct student interventions 
in the Round 1 colleges were designed to change classroom instruction; most of the 
interventions involved either student support services or academic assistance designed to 
support instruction in the classroom. Second, across the Round 1 colleges there was a trade-off 
between the scale and intensity of implemented ATD strategies. Low-intensity strategies, which 
tended to be support services such as orientation and advising programs, were more likely to be 
implemented at scale than were high-intensity strategies such as curricular reforms, 
supplemental instruction, and learning communities, which generally “touched” relatively small 
numbers of students. Third, while some of the colleges did implement strategies that touched 
large numbers of students, the majority of all direct interventions at the Round 1 colleges 
remained small in scale. These findings show the difficulty of implementing interventions on a 
scale sufficient to reach substantial numbers of students. Yet, as MDRC and CCRC pointed out 
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in the 2011 report on the Round 1 institutions, unless colleges can implement high-intensity 
instructional reforms at scale, they are unlikely to be able to increase completion and success 
rates for substantial numbers of students. 

The Round 1 colleges and the Washington State colleges targeted similar student 
populations for the majority of their improvement strategies. Both groups of colleges focused on 
developmental education for the majority of their interventions. At the Round 1 colleges, nearly 
half of all strategies involved work on this area, and at the Washington colleges the proportion 
was even higher (56 percent). Furthermore, in both groups, strategies targeting first-year 
students were the second most common after developmental education. 

However, one notable difference between the Round 1 and Washington colleges was 
that the Washington colleges focused their ATD activities on instruction to a greater extent than 
did the Round 1 colleges. Among the Round 1 colleges, instructional changes (including both 
reforms to curriculum and instruction and to instructional supports) made up 52 percent of all 
direct student success strategies. In Washington, instructional changes accounted for 70 percent 
of all direct strategies (see Figure ES.l). In this, the Washington colleges may have benefited 


Achieving the Dream Colleges in Washington State 
Figure ES.l 

Proportion of Student Success Strategies by Type Used at 
Round 1 and Washington ATD Colleges 


Round 1 


Washington 


Curriculum 
and Instruction 
27 % 

Instructional 

Supports 

25 % 


Student 

Support 

Services 

48 % 


Curriculum 

and 

Instruction 

41 % 


Student 

Support 

Services 

30 % 


Instructional 

Supports 

29 % 
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from the experiences of the Round 1 colleges and learned about successful approaches for 
changing instructional practices. 

Another difference between the Round 1 and Washington colleges was the scale of 
student interventions. Figure ES.2 shows that 31 percent of strategies implemented by the 
Round 1 colleges were operating at scale (defined as affecting at least 25 percent of the target 
student population), compared with 43 percent of strategies at the Washington colleges. And in 
contrast to the Round 1 colleges, the large-scale strategies at Washington colleges were not 
mostly limited to student services. Over half of the strategies that had reached a substantial 
number of students in Washington involved changes to instructional practices. 


Achieving the Dream Colleges in Washington State 
Figure ES.2 

Proportion of Strategies Operating at Small, Medium, or Large Scale 
at Round 1 and Washington ATD Colleges 


Round 1 


Washington 


Large 

31% 


Medium 

17% 


Small 

52% 


Small 

43% 39% 


Medium 

18% 


The research literature on organizational effectiveness suggests that no one innovative 
practice or even set of discrete practices is sufficient to improve organizational performance. 
For community colleges, the implication is that to achieve substantial improvements in student 
outcomes, they will need to make systemic changes in programs and services. From this 
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perspective, rather than test and scale up discrete interventions, colleges need to redesign major 
programs and functions at scale. 

While we saw some cases among the Round 1 colleges of institutions making systemic 
changes to programs and services, this was more prevalent among the Washington colleges. For 
example, one college developed a comprehensive approach to helping first-year students 
navigate from matriculation to the declaration of a program of study, with the goal of helping 
them obtain 30 credits by the end of the first year. The multi-pronged approach included early 
assessment, mandatory orientation, a redesigned student success course, and intrusive advising 
for students, along with a locally developed software application for tracking student 
progression and improving advising. As a result of integrating interventions into a more 
comprehensive model that supports students along a pathway from intake to the accumulation 
of 30 college credits, the college has seen some initial success in improving the experience of 
first-year students. 

On the instructional front, one college transformed its developmental math sequence to 
make the math curriculum more relevant to students’ academic pathways. In addition to the 
curricular reform, the strategy reduced the length of the developmental math sequence from 
three terms to two terms for students not pursuing STEM programs of study. Administrators 
reported that because the entire developmental math structure had changed, faculty became 
engaged in deeper conversations about math content and placement procedures more generally. 
At two other colleges, faculty led efforts to spread effective teaching methods across academic 
programs. 

These cases suggest that the engagement of faculty in the knowledge-building and 
ownership of reform efforts is an important component of implementing changes in classroom 
practice at scale. They also suggest that without the active support of administrators to facilitate 
communication and spread reform efforts, broad faculty engagement and scaling up efforts are 
likely to be more difficult to achieve (as we saw in at least one of the other colleges). The 
Washington colleges that were most likely to make systemic changes by redesigning programs 
and services and by supporting the diffusion of instructional innovations were those that we 
found to be further along in implementing the ATD culture of evidence principles. This finding 
supports the idea, central to the Achieving the Dream theory of change, that building a culture 
of evidence creates the organizational conditions for reforming academics and student services 
to better serve students. 

The approach taken by these colleges was essentially to innovate at scale rather than to 
scale innovations. Although they did experiment with different approaches, the colleges tended 
to redesign programs and services college-wide. Such innovations are easier for colleges to 
sustain since they become part of the way colleges operate. We hypothesize that the colleges 
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that were able to innovate at scale are those that will see the greatest improvements in 
performance. 


Data Use by Faculty, Administrators, and Staff 

We conducted surveys of the use of data by faculty and administrators at the 
Washington colleges in two waves, one in 2007-08, when the colleges first began to implement 
their ATD work, and one in 2011, near the end of the five-year implementation phase. Based on 
the surveys, we found some positive changes in the extent of use of data over time by faculty 
for teaching-related purposes, particularly at three of the colleges. We observed no changes, 
however, in the frequency with which faculty used data on academic achievement gaps between 
students of different races/ethnicities and income levels. 

Faculty at three of the colleges perceived an increase in the use of data by their 
academic departments, particularly for curricular decisions and identifying courses with high 
failure rates. Yet use of data by administrative departments at the colleges declined between the 
two waves of the survey. This suggests that use of data for planning purposes waned following 
the first year or two of participation in Achieving the Dream, when colleges were strongly 
encouraged and were given help to analyze data on student success. 

These findings suggest that some positive changes in attitudes toward and use of data 
took place during the initial implementation phase of the first six Washington ATD colleges. 
Moreover, the colleges that made the most progress in implementing practices that reflect the 
ATD culture of evidence principles were more likely to show evidence of increased data use. 
However, there were perhaps not as many improvements as we expected at the outset of this 
initiative, which has among its core principles the creation of a culture of evidence to improve 
student outcomes. Although we are encouraged by the generally positive changes that occurred, 
these patterns of improved data use varied across colleges and within colleges between faculty 
and administrators. 


Student Outcome Trends 

Using student-level data from the entering fall cohorts of 2003 through 2009, we 
examined average initiative-wide outcomes across the six Washington colleges on five key 
ATD performance indicators, measured two years after the students entered college: (1) 
attempting and completing developmental education (remedial) courses in English and 
mathematics, (2) attempting and completing the first college-level course (gatekeeper) in 
English and mathematics, (3) attempting and completing credits, (4) persisting fall-to-fall and 
year-to-year, and (5) earning certificates or degrees. Each cohort was followed for a two-year 
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period. For example, the 2009 cohort was tracked through the 2009-10 and 2010-11 academic 
years. 


The main findings from the student outcomes analysis are: 

• The average student outcomes across the six colleges appear largely 
unchanged after the colleges began implementing Achieving the Dream. 

• The initiative does not appear to have reduced the achievement gap along 
racial or economic lines. The gap between students who received Pell 
grants (a measure of low-income status) and those who did not does 
appear to decrease over students’ academic careers, but this was also true 
prior to implementation of Achieving the Dream. 

• There do not appear to be greater gains associated with Achieving the 
Dream for some colleges compared with others. 

These findings should be interpreted cautiously. The colleges implemented a wide 
variety of student success strategies, and their strategies seem to be reaching more students than 
in earlier rounds of the initiative. In this sense, the colleges appear to have built on the 
experiences of previous rounds of Achieving the Dream, and in recent years, they have been 
moving closer to affecting large portions of their student populations. With respect to the 2007 
through 2009 cohorts included in this analysis, however, the Washington colleges were still 
reaching relatively small proportions of students through their new strategies. Moreover, few of 
the colleges reached scale with a comprehensive set of strategies that touch students across the 
full range of their experiences, even in more targeted areas of the student experience such as 
intake and remediation, where most of the ATD strategies were focused. It may be that the 
colleges positively affected students on a smaller scale than these analyses could detect, or that 
they positively impacted larger numbers of students in subsequent years. 


Benefits of Participating in ATD 

We asked those we interviewed at the colleges what they saw as the main benefits of 
participating in ATD. Five of the six colleges indicated that they benefited in substantial ways 
from participating in the initiative. Table ES.3 lists the main benefits and indicates the number 
of colleges mentioning each. 
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Achieving the Dream Colleges in Washington State 


Table ES.3 

Perceived Benefits of ATD for Organizational Practice at the Washington Colleges 


Type of Benefit 

Specific Benefit 

Number of 
Colleges 

Institutional Research 

• Greater access to data and stronger capacity to use 

5 

and Data Use 

data in decision-making processes 



• Increased use of data to improve college programs and 
policies 

4 


• Use of data for greater transparency in resource 
allocations/budget decisions 

2 


• Strengthened grant writing capacity 

2 

Student Success 

• Systematic approach to developing strategies based on 

4 

Strategies 

student outcomes data 



• Impetus to develop/expand strategies 

2 

Initiative Alignment 

• Synergy with new accreditation process and with WA 
Student Achievement Initiative 

6 


Respondents at five of the six colleges indicated that their organization has benefited 
from an increased access to and use of data and research on student outcomes. Three colleges 
mentioned that one benefit of increased IR capacity was a strengthened capacity to pursue grant 
funding. 

Three of the colleges credited their participation in ATD with strengthening the process 
by which they identify ways to improve programs and services, specifically by improving the 
quality of data for making decisions about what to improve and how. All six colleges said that 
ATD helped them better align efforts to improve student completion, particularly those related 
to meeting the new accreditation standards. 

Most of the colleges found their ATD coaches (a leadership coach and a data coach) to 
be helpful. There was broad agreement among those we interviewed at one college that the both 
the leadership coach and the data coach challenged the college to improve and, as one 
respondent said, “kept us accountable.” In the last couple years, both coaches at the college 
have sought to focus college leaders on figuring how to sustain both successful strategies and 
the overall work on improving outcomes after the initial five years in ATD. Respondents at 
another college said that their data coach in particular helped to frame the data analyses by 
asking the right questions and facilitating the process of inquiry and discussions of data by the 
team. Leaders at a third college said that they would only continue to participate in ATD 
beyond the initial period if they could have the same leadership coach and data coach. 
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Lessons for Other Colleges 

Based on our in-depth analysis of the experience of the six original Washington State 
ATD colleges during their first five years in the initiative, we highlight several lessons for the 
new Washington ATD colleges and others elsewhere seeking to implement similar reforms. 

Redesign Programs and Supports at Scale 

To achieve substantial improvements in student outcomes, colleges need to make 
systemic changes in programs and services — no single innovation or even set of interventions 
will be sufficient to “move the needle” on student completion. From this perspective, rather than 
test and scale up discrete interventions, colleges should redesign major programs and functions 
at scale. Rather than try to bring to scale discrete interventions, research on high performance 
organizations both in and outside of education indicates that improving organizational 
performance generally requires systemic changes in practice. 

Some of the Washington colleges did seek to move beyond implementing discrete 
interventions to make systemic changes in programs and practices, and several of the colleges 
succeeded in this. Some of these same colleges also succeeded in promoting diffusion of 
instructional innovations. In those cases, the colleges seemed to provide a balance between 
empowering faculty and leaving them alone to try out innovations while at the same time 
providing support for professional development. 

The Washington colleges that were most likely to make systemic changes by 
redesigning programs and services and supporting the diffusion of instructional innovations 
were those that we found to be further along in implementing the ATD culture of evidence 
principles. This finding supports the idea, central to the Achieving the Dream theory of change, 
that building a culture of evidence creates the organizational conditions for reforming 
academics and student services to better serve students. 

Involve Faculty From the Start in Leading Efforts to Rethink 
Curriculum and Instruction 

Community colleges are unlikely to see substantial improvements in student completion 
and learning unless there are major changes in classroom practice. Classroom practices are not 
going to change unless the faculty are convinced of the need to do so and are supported to 
explore and implement changes. It is critical, therefore, that colleges seeking to improve 
programs and instruction involve faculty from the start in leading such reforms. 

Two or three of the Washington colleges succeeded in making major changes in 
instructional practice. Faculty at one college revamped its entire developmental mathematics 
curriculum; at another, a group of faculty spearheaded widespread adoption across departments 
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of a tested methodology for improving students’ comprehension of texts. Colleges that want to 
engage faculty in reforms should do so early on, being clear about the desired goals but 
empowering them to decide, in conjunction with their student support services colleagues, how 
best to achieve them. Colleges should also support the spread of innovative practices by 
providing the infrastructure and opportunities for faculty and staff across the institution to share 
their knowledge and experience. 

Strengthen Pathways Into and Through Programs of Study 

The efforts by the Washington colleges to redesign programs and services were mostly 
confined to the intake process and developmental education. Over time, as these colleges 
succeed in implementing these reforms at scale, we hypothesize that these changes will lead to 
improved success in developmental courses and progression to college-level gatekeeper 
courses. Most of these reforms were implemented at scale after 2009, so their effects are not 
likely to be reflected in the data we have on student progression, which only go up to the 2009 
first-time fall cohort. In the longer term, however, without systemic changes to colleges’ 
academic programs, we do not see how “front-end” reforms of the sort mainly implemented by 
the Washington colleges alone can substantially increase student completion and success. 

Research suggests that a similar systemic reform may also need to be applied to 
college-level programs to ensure that they are well structured and aligned with requirements for 
success in further education and, for career-technical education programs, employment. With a 
handful of exceptions, the focus of instructional reforms in the Washington colleges was on 
improving student success in developmental education and introductory college-level math and 
English. Far less attention was paid to improving student readiness and success in other 
introductory college courses that serve as gatekeepers to programs in other fields, such as social 
and behavioral sciences, biological and physical sciences, and business. Taking steps to 
strengthen programs of study in these fields would also have the effect of broadening 
involvement by faculty in areas other than English, math, and (in colleges where it is a separate 
department) developmental education. Colleges also need to strengthen mechanisms to help 
students choose from among the many programs offered. The work by one college (WA-C) to 
restructure the intake and advising functions to help students choose a program and earn college 
credits in it as soon as possible offers a promising model. 
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Use a Common Set of Performance Measures to Focus Strategy 
Development and Promote Sharing 

Each of the Washington colleges has developed clear goals for student success with 
accompanying metrics, spurred in part by the new accreditation standards of the Northwest 
Commission on Colleges and Universities that emphasize the use of metrics and evidence in 
planning and improvement. Each college has developed its own set of goals and measures. 
While some of the colleges have borrowed from ATD performance measures, none have 
adopted them as a set. 

While it makes sense for colleges to adopt their own performance measures, from the 
standpoint of ATD it is problematic that there is not a common set of metrics that colleges 
across the initiative recognize and pay attention to. Unless colleges focus their reforms on 
particular metrics, it is unlikely that they will develop innovations that will improve their 
performance on those metrics. Also, without shared performance measures across colleges, it is 
difficult for the initiative and colleges participating in it to identify colleges that do improve 
their performance. 

Therefore, we recommend that Achieving the Dream encourage colleges across the 
initiative to adopt a common set of performance measures for use in developing and examining 
the effectiveness over time of their own strategies and in comparing colleges on particular 
measures for the purpose of identifying colleges that have improved their performance over 
time. 


Looking Ahead 

Making systemic changes in practice in the ways that most of the original Washington 
State ATD colleges already have begun to do and in the other ways recommended here requires 
resources — resources for new and strengthened information systems, for professional 
development and training, and for coordination. The Washington ATD colleges have shown 
that even during a period of rising enrollments and dramatic cuts in funding, they have been 
willing to make major changes in practice. They have done so by reallocating their scarce 
resources from practices that are not effective to those that are thought to be so. The strong 
commitment to student success of these colleges, combined both with the interest of other 
colleges in the state in joining ATD and the willingness of College Spark and Lumina 
Foundation to support them, makes us optimistic that Washington’ s community and technical 
colleges, along with the State Board for Community and Technical Colleges, will find ways in 
this difficult economic climate to make changes that are necessary to improve student 
outcomes. 
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Chapter 1 

Introduction 


This report presents findings from research conducted by the Community College 
Research Center (CCRC) and MDRC in spring 2011 on the implementation work of 
community and technical colleges in Washington State that are participating in Achieving the 
Dream (ATD). The findings are based on field research, a survey of data use by college 
personnel, and an analysis of trends in academic performance. The report compares the findings 
to those of an earlier “baseline” examination of the initial work and early progress of 
Washington and Pennsylvania ATD colleges that CCRC and MDRC conducted in spring 2008 
(Jenkins, Ellwein, Wachen, Kerrigan, & Cho, 2009). It also compares the progress of the 
Washington colleges after their first five years in ATD with the progress of the first round of 
colleges to join the initiative, which was examined in a 2011 MDRC-CCRC report (Rutschow 
et al., 2011). 


Overview of Achieving the Dream 

Achieving the Dream is a national nonprofit dedicated to contributing to the nationwide 
effort to increase the number of students, particularly low-income students and students of 
color, who earn postsecondary credentials with marketplace value. ATD was conceived in 2004 
as a reform movement by Lumina Foundation for Education and seven partner organizations: 
the American Association of Community Colleges; the Community College Leadership 
Program at The University of Texas at Austin; the Community College Research Center at 
Teachers College, Columbia University; Jobs for the Future; MDC; MDRC; and Public 
Agenda. In 2010, Achieving the Dream became an independent nonprofit organization. 1 

ATD and its partner organizations provide participating colleges with technical support 
to help them actualize their reform plans. Each college works with an ATD leadership coach 
and data coach (also called a data facilitator) who provide technical assistance for ATD work 
throughout a one-year planning period and a four-year implementation period. The colleges are 
encouraged to develop ATD core teams and data teams consisting of representatives from their 
various divisions to address the work of ATD generally and the efforts to increase data and 
research capacity at the institution specifically. 


1 For additional information about Achieving the Dream, see the initiative’s website, 
www.achievingthedream.org. 
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Currently more than 150 institutions in 30 states and the District of Columbia 
participate in ATD. Colleges joined the initiative in different years, with the first group of 
participating colleges (located in Florida, New Mexico, North Carolina, Texas, and Virginia) 
joining in the 2004-2005 academic year. These institutions are referred to as the Round 1 
colleges. Six colleges in Washington State were part of the third round of colleges to join ATD 
in the 2006-2007 academic year, and they are covered in this report. The Washington colleges 
each received a four-year grant of $400,000 from College Spark Washington during the 
implementation period. 


The Achieving the Dream Culture of Evidence Principles 

ATD seeks to improve student success on a substantial scale by fundamentally 
changing the way that community colleges operate. To accomplish this goal, ATD promotes a 
comprehensive, evidence-based reform model. Colleges develop a “culture of evidence” in 
which data on student progression and outcomes are used to make decisions about the design 
and implementation of programs and services. Four main principles govern the ATD culture of 
evidence model for institutional improvement: (1) committed leadership, (2) use of evidence to 
improve programs and services, (3) broad engagement of stakeholders, and (4) systemic 
institutional improvement. 2 

Committed Leadership 

Committed leadership is essential to effect systemic change that will improve student 
outcomes and narrow gaps in achievement between students of different races/ethnicities and 
income levels. The college president and other senior administrators marshal support across the 
institution for an overall vision of student success and for specific efforts to reform policies, 
programs, and services. Board members and faculty leaders help to support this vision. 

Use of Evidence to Improve Programs and Services 

The use of data and research to drive institutional improvement is central to the ATD 
culture of evidence model. Colleges use data on student progression and outcomes to design, 
deliver, and fund strategies to improve programs and services and to evaluate the effectiveness 
of those strategies. Data are also used to identify and address achievement gaps. 


2 For a more in-depth description of the model, see Rutschow et al., 201 1, pp. 9-13. 
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Broad Engagement 

The widespread engagement of stakeholders across the institution is necessary to bring 
about sustainable institutional reform. Faculty, staff, administrators, students, and the larger 
community all are engaged in efforts to enhance student success, and they work collaboratively 
to develop, evaluate, and improve student success interventions. 

Systemic Institutional Improvement 

Colleges build an infrastructure to support continuous, systemic improvement. They 
make substantial changes in policies, organizational infrastructure, and budget allocations that 
are evidence-based and outcomes-driven. Professional development opportunities for faculty 
and staff are provided to reinforce student success. 


Achieving the Dream Colleges in Washington State 

Six colleges in Washington State joined Achieving the Dream during the 2006-2007 
academic year. They were selected because they enrolled the highest proportions of low-income 
students and/or students of color among the 34 community and technical colleges in the state 
system. The six colleges, along with a group of Pennsylvania colleges that also joined that year, 
were the third group of colleges to join the initiative and therefore are referred to as the Round 3 
colleges. Table 1.1 identifies the Washington colleges and presents enrollment information from 
2006-2007 and 2010-2011. This report examines the implementation work at the Washington 
colleges as of spring 2011, five years after the colleges first joined Achieving the Dream. 
Throughout the rest of this report, the colleges are not identified by name in order to optimize 
learning from their experiences. 
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Achieving the Dream Colleges in Washington State 


Table 1.1 

Achieving the Dream Colleges in Washington State 


College 

Location 

2006-2007 

Enrollment 

(FTE) 

2010-2011 

Enrollment 

(FTE) 

Big Bend 

Moses Lake 

1,487 

1,878 

Highline 

Des Moines 

4,687 

6,171 

Renton Technical 

Renton 

2,166 

2,908 

Seattle Central 

Seattle 

5,138 

6,154 

Tacoma 

Tacoma 

4,758 

6,360 

Yakima Valley 

Yakima 

3,483 

3,771 


SOURCE: U.S. Department of Education, National Center for Education Statistics, Integrated 
Postsecondary Education Data System. 


As is evident from Table 1.1, the first five years of the Washington colleges’ 
involvement in ATD was a period of rapid enrollment growth, brought on at least in part by the 
Great Recession, when many workers returned to college to improve their skills and earn 
marketable credentials. At the same time — also as a result of the recession — the Washington 
community and technical colleges suffered major cuts in state funding, which, together with 
tuition, is their main source of revenue. From FY 2009 to FY 2013, state funding for the 
community and technical colleges declined by 23 percent. While the legislature allowed tuition 
increases, they were not sufficient to cover the losses from reduced state funding. With 
increased revenues from tuition factored in, the net cut to the colleges averaged about 7 percent. 

Even as Washington state policy makers were forced to cut funding for community and 
technical colleges, like their peers across the country, they increased pressure on public colleges 
and universities to improve outcomes. Partly as a response, in 2007 the State Board for 
Community and Technical Colleges adopted a performance funding policy called the Student 
Achievement Initiative, through which colleges receive funding for improving student 
attainment of “achievement points,” earned for both completion of credentials and achievement 
of intermediate milestones such as taking and passing college-level math or accumulating a 
substantial number of college credits. 

As discussed in this report, most of the Washington ATD colleges saw Achieving the 
Dream as offering a framework and general approach for bringing about the sorts of changes in 
practice needed to meet the increasing demands for outcomes and accountability. The emphasis 
of ATD on building a culture of evidence was also consistent with the new accreditation 
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standards that place much greater emphasis on the use of evidence in planning and decision 
making, which the Northwest Commission on Colleges and Universities adopted during this 
period. 


In its goals and approach, Achieving the Dream was consistent with the direction in 
which the Washington colleges were feeling pressure to go and indeed were already moving in. 
However, in its emphasis on using evidence for improvement, in its focus on reducing 
achieving gaps among students, and in other respects, ATD expects colleges to operate in some 
ways that are very different from what they are accustomed. Making substantial changes in 
practice requires resources for implementing new approaches and managing the change process, 
and it also requires the broad engagement of faculty and staff. In examining the efforts of the 
Washington colleges to change how they managed and delivered programs and services 
according to the ATD model, it is important to keep in mind that the colleges were operating in 
a context of rising enrollments and declining state funding. These factors made the task of 
bringing about systemic changes — difficult in any circumstances — even more challenging for 
the Washington colleges during their first five years in the initiative. 


Research Questions 

As part of the overall evaluation of Achieving the Dream, Lumina Foundation funded 
an evaluation of the Round 3 Washington colleges to examine the progress they made in 
adopting the ATD culture of evidence model and in improving student success. Thus, this report 
on the Washington colleges examines the following research questions: 

• Have the colleges made progress in building a culture of evidence by 
implementing the ATD principles of institutional improvement? 

• What student success strategies did the colleges implement under ATD, and 
which of these are being implemented at scale? 

• Has involvement in ATD changed the way that faculty, staff, and 
administrators use data? If so, in what ways? 

• What have been the trends in student achievement among the Washington 
ATD colleges? Is there evidence that student progression and outcomes 
improved during the period after the Washington colleges joined ATD? 

Lumina and other ATD funders were particularly interested in learning whether the 
Washington colleges, which joined the initiative when it was more fully developed, progressed 
faster than the colleges that joined in Round 1 . So another focus of this report is a comparison of 
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the progress of the Washington colleges over the first five years of their involvement in the 
initiative with that of the Round 1 ATD colleges. 


Research Methods 

We collected data for this evaluation through a review of the Achieving the Dream 
internal reports, field research, a survey of data use by faculty and administrators, and student 
records. 


Review of Achieving the Dream Internal Reports 

Colleges participating in Achieving the Dream are required to submit annual narrative 
reports to the initiative. They describe each college’s progress in implementing the ATD 
principles of institutional improvement. In addition, the ATD leadership coach and data coach 
assigned to each college complete assessment and feedback reports several times a year during 
the implementation period to assess the progress made at each college. Researchers reviewed 
these documents for each college in Washington prior to conducting the field research. 

Field Research 

The CCRC-MDRC research team conducted a two-day site visit at each the six 
Washington colleges in April and May 2011. During these visits, researchers conducted 
interviews with the college president, vice presidents or deans of instruction and student 
services, institutional researchers, the Achieving the Dream coordinator, and board members. 
Group interviews were also conducted with small groups of faculty members, student services 
staff, and students directly involved with Achieving the Dream efforts, as well as with faculty 
and staff representatives not directly involved in the initiative. Researchers used a structured 
interview protocol to maintain consistency in questioning across institutions and contacted 
college administrators and program leaders to obtain additional information after the site visits 
had been conducted. 3 The results of the spring 2011 field research were compared with those of 
site visits conducted in spring 2008 to assess changes in the policies and practices of the 
colleges over that period. 

Survey of Data Use 

A panel survey of data use by faculty and administrators at the six Washington colleges 
was administered in both 2007 and 2011. The CCRC-MDRC research team modified the 


3 The framework used to develop the interview protocols is presented in Appendix A. It is based on 
the framework used in the baseline field research on the Washington ATD colleges and is the same as the 
one used in the second wave of field research on the Round 1 colleges. 
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surveys in 2010 prior to the second administration to capture additional information and details. 
The sampling frame included all full-time faculty and administrators at the director level or 
above. Student services staff were also included in the 2011 survey. 

Student Records 

The colleges submitted student records to the Achieving the Dream database, managed 
by JBL Associates (a higher education consulting firm) under contract with CCRC. These data 
contain information on full- and part-time students entering the colleges each fall and their 
academic outcomes at the colleges in subsequent years. Measures of student outcomes include 
referral to and completion of developmental sequences in English and math, completion of 
gatekeeper courses in English and math, attempted and completed credits, persistence from fall 
semester to fall semester, and credential attainment. These data were collected for cohorts that 
entered from fall 2003 through fall 2009. 


Organization of the Report 

This report examines the progress of six Washington Round 3 ATD colleges in building 
a culture of evidence and improving student outcomes. Chapter 2 assesses the colleges’ 
progress in building a culture of evidence by implementing the four main principles of the ATD 
model for institutional improvement. Chapter 3 examines the strategies for improving student 
success implemented by the colleges as part of ATD and compares them with the strategies 
developed by the first round of colleges to join the initiative. Chapter 4 describes the patterns 
and extent of data use by college faculty and administrators. Chapter 5 analyzes trends in the 
academic performance of students at the six colleges using student record data collected through 
the initiative. Chapter 6 concludes the report by discussing the benefits of participating in 
Achieving the Dream on the Washington colleges and provides recommendations for enhancing 
the impact of the initiative both in Washington State and beyond. 
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Chapter 2 

Progress Toward Building a Culture of Evidence 


This chapter assesses the progress of the six Washington colleges in building a culture 
of evidence — that is, in actualizing the four main principles of the Achieving the Dream 
culture of evidence model for institutional improvement — in the three years between spring 
2008 and spring 2011. The principles, described in detail in Chapter 1, are: (1) commitment of 
leadership to improving student outcomes and equity, (2) use of evidence to improve student 
success, (3) broad engagement of stakeholders in the improvement process, and (4) continuous, 
systemic institutional improvement. 

First, we present an overall assessment of the progress that colleges have made in 
implementing the ATD culture of evidence principles by spring 2011. We then explore in more 
depth the colleges’ efforts to adopt practices that reflect the culture of evidence principles and 
describe barriers faced by the colleges in the process. Next we summarize the main changes in 
organizational practice at each college that resulted directly from the college’s participation in 
ATD. Finally, we compare the progress in building a culture of evidence at the Washington 
colleges with the first group of colleges to join the initiative. 

The key findings of this chapter are the following: 

• Five colleges made progress toward building a culture of evidence; two made 
substantial progress. 

• Leaders at only three colleges made a strong commitment to addressing gaps 
in achievement by students of color and low-income students, despite the 
focus of ATD on equity. 

• Three colleges added institutional research (IR) staff, and four colleges 
supplemented the work of IR staff with a “data team” comprised of faculty, 
staff, and administrators. 

• The six Washington colleges, as a group, went at least as far as the Round 1 
colleges in building a culture of evidence, although it is difficult to compare 
them with the 26 first-round ATD institutions. 


Overall Assessment of Progress 

In spring 2008, when CCRC and MDRC conducted a first round of field research at the 
Washington colleges, two of the colleges were making progress toward implementing the ATD 
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culture of evidence principles; both had begun to move in this direction before joining ATD, but 
participation in ATD helped give their efforts momentum. Two other colleges had made some 
progress, although many obstacles remained. The remaining two colleges had made very little 
progress, mainly because of a lack of leadership commitment or, in one case, turnover in 
relevant top leadership. 

Table 2.1 presents an overall assessment of the extent to which the Washington colleges 
had implemented the ATD culture of evidence principles in spring 2008 and spring 2011, and it 
indicates the extent of progress that each college made in that period. This assessment is based 
on a framework and rubric that the CCRC-MDRC research team used to evaluate the extent to 
which each college had implemented the ATD principles. The protocol was structured to assess 
each college on 34 dimensions of practice aligned with the four ATD principles. For each 
dimension, the college received a rating from 1 (little or none) to 5 (high level/a lot). The 
framework used to develop the protocol is presented in Appendix A. It is based on the 
framework used in the baseline field research on the Washington State ATD colleges and is 
essentially the same as the one used in the second wave of field research on the Round 1 
colleges. Detailed memos written after each visit were also analyzed to inform each college’ s 
overall rating on the level of implementation. 

All but one of the colleges made at least some progress toward building a culture of 
evidence according to the ATD principles. Two colleges — WA-C and WA-D — made 
substantial progress, moving from a low level of implementation to a high level over the first 
five years of their involvement with the initiative. 


Achieving the Dream Colleges in Washington State 
Table 2.1 

Level of Implementation of ATD Culture of Evidence Principles 
at the Washington Colleges: A Comparison of Spring 2008 and Spring 2011 


College 

Spring 2008 

Spring 2011 

Increase in Levels 

WA-A 

High 

Very high 

+ 1 

WA-B 

Medium 

High 

+ 1 

WA-C 

Low 

High 

+2 

WA-D 

Low 

High 

+2 

WA-E 

Very low 

Low 

+ 1 

WA-F 

Very low 

Very low 

0 
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Successes and Challenges in Actualizing the Culture of Evidence 
Principles 

This section examines the extent to which the Washington colleges have adopted 
practices that reflect the four ATD culture of evidence principles and identifies barriers that the 
colleges faced in the process. Findings are based on assessments made by CCRC and MDRC 
researchers during our spring 2011 field research using a framework and protocol similar to 
those used in our field research in 2008. (The tables illustrating the findings appear after the 
discussions of the four principles.) 

Committed Leadership 

Table 2.2a summarizes the extent to which the practices of each college reflect the first 
principle of the ATD culture of evidence model: leadership committed to improving student 
outcomes and narrowing achievement gaps. 

President and Senior Administrators 

Leadership is critical to building a culture of evidence, and the colleges that were 
further along in implementing the ATD principles had higher levels of leadership commitment. 
At three of the colleges — WA-A, WA-B, and WA-C — the leaders were committed to 
evidence -based improvement from the start. The president at WA-D was initially a skeptic, 
largely because of the perception that data previously collected and reported were unreliable; 
however, under Achieving the Dream this president became a champion of data-based decision 
making as a result of the college’s success in building an institutional research and effectiveness 
function that produces data the president and others on campus trust. 

The president and top academic administrators at WA-E were not convinced that the 
Achieving the Dream approach was the best way to improve student outcomes. Instead, they 
implemented an academic master plan with strategies for improving the quality of teaching and 
learning at the college. At WA-F, turnover among the college’s top leadership made it difficult 
to sustain a focus on student success. The incoming president and his leadership team were 
primarily concerned with improving the college’s image in the community and developing 
alternative sources of funding. 

Faculty 

The faculty members we interviewed at all six colleges were interested in improving 
student outcomes, but in only three of the colleges were they working together on a widespread 
basis to improve student success. When WA-A joined ATD, the college already had a history of 
faculty working together to develop innovations. At WA-B, the impetus to join ATD came from 
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the college leadership, but faculty — in part as a response to what was perceived as an overly 
top-down approach to managing the initiative — later took the lead in developing and 
implementing instructional innovations, particularly the Reading Apprenticeship strategy, 
which has since been embraced widely by faculty across divisions. (The work of the faculty in 
implementing Reading Apprenticeship at WA-B is profiled in the next chapter.) At WA-F, 
ATD project leaders latched onto the efforts of a group of entrepreneurial faculty to enrich 
teaching and learning through integrative assignments. However, this innovation and others we 
heard about were not widely propagated among faculty across departments. 

Trustees 

We interviewed at least one board member at each college. Generally, they were 
informed about ATD and its purposes and had received periodic reports on their college’s 
progress on the initiative. At four colleges — WA-A, WA-B, WA-C, and WA-D — we heard 
from trustees that college leadership had improved the quantity and quality of data on student 
outcomes since their institutions first joined the initiative. For example, a board member at WA- 
C said that under ATD the college moved from a focus on “just access” to “helping students on 
a path to completion.” This board member indicated that the board generally was attuned to 
using data for decision making but, given recent funding cuts, data were particularly important 
for helping decide what programs and services are critical to preserve. 

In June 2011, after we had conducted our field research at the colleges, presidents and 
trustees from all the Washington community and technical colleges participated in the 
Governance Institute for Student Success (GISS). At this two-day institute, sponsored by the 
Association of Community College Trustees and the Community College Leadership Program 
at the University of Texas at Austin, trustees and presidents were presented with detailed reports 
on student outcomes at their college. The reports were based on performance metrics developed 
by the state’s Research and Planning Commission (the organization of two-year college 
institutional research directors) that measured for each college rates of progression and 
completion over time of cohorts of students broken out by the type of program they were 
enrolled in (e.g., basic skills, transfer, or workforce) and student characteristics. The institute 
was led by Byron McClenney, one of the founding partners of Achieving the Dream who leads 
the coaching aspect of the initiative, so the reform principles used reflect those of ATD. 
Learning events like this promise to raise awareness among college trustees about the 
completion agenda and the importance of using data to improve student outcomes. 

Commitment to Closing Achievement Gaps 

In the initial analyses they conducted during the planning year, most of the colleges 
identified gaps in achievement between White students and students of color. Since joining 


12 



ATD, five of the six colleges have experienced changes in the demographics of their 
surrounding communities, particularly increases in the number of immigrant and low-income 
residents. These changes have resulted in increasing numbers of disadvantaged students on their 
campuses. 

Serving disadvantaged students is challenging in any period, but especially in times of 
budget cuts, when services for all students are cut back. Long before joining ATD, WA-B 
embraced the mission of serving the local community, with a special focus on helping provide 
access to economic opportunity for underserved residents, especially immigrants. The college 
considered its adult basic skills programs to be “bridges” to college-level training programs. 
However, recent research conducted by the college also revealed substantial gaps in 
achievement levels between White students and African American students. College leaders say 
they would like to do more to address this gap, but budget cuts have forced them to cut back on 
student services for all students. Some of the other colleges faced a similar challenge. 

Despite challenges such as these, leaders at three of the colleges have made a strong 
commitment to addressing the issue of achievement gaps. The president of WA-C has made it a 
priority to address disparities in achievement by race and class. Both within the college and with 
the local K-12 schools, the president has led sometimes difficult conversations about the 
achievement gap problem and how to deal with it. On campus, faculty indicated that, after an 
initial period of discomfort, more of their colleagues were open to having these conversations. 
As part of ATD, the college implemented several strategies to improve outcomes for students of 
color. College administrators claim that these efforts have helped to narrow the achievement 
gap between African American and White students on campus, although they admit they have 
made less progress on reducing the achievement gap between their Hispanic and White student 
populations. 

At WA-A, the initial ATD data analyses identified three groups of students that were 
falling behind: immigrants in basic skills courses, students of color (in math especially), and 
displaced workers seeking retraining for new jobs. The college designed its ATD strategies to 
help improve achievement among students in these groups. College leaders said that they have 
made progress in reducing achievement gaps, particularly for students of color in the 
developmental math sequence and new immigrants in the English as a Second Language (ESL) 
sequence. 

At WA-D, which has seen growth in the number of low-income Hispanic students in its 
service region, difficult conversations about achievement gaps between Hispanic and White 
students were hampered by a lack of data that the stakeholders at the colleges could trust. After 
the college established an institutional research function that produced reliable, trustworthy data 
(see the case study below, Box 2.1, Building an Effective Home-Grown Institutional 
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Effectiveness Function), discussions about the achievement gaps could and did take place. As a 
result, the college’ s administrative council and board of trustees, as part of the its strategic plan, 
approved two “core theme objectives” designed to improve equity of access to the transfer and 
professional-technical programs of the college, and seven objectives designed to improve equity 
of student achievement. The college’s institutional effectiveness team regularly reports on the 
college’s progress toward these objectives. Although it too has experienced an increase in the 
number of low-income Hispanic students in its community and the college’s research shows 
gaps in achievement between Hispanic and White students, WA-E did not make equity of 
achievement a strategic institutional focus. Minority student services staff lobbied the college to 
do more, but most of the college’s efforts relied on their initiative; the college’s overall response 
to the problem was limited. 

Use of Data and Research for Improvement 

Table 2.2b summarizes our assessment of the extent to which the practices of each 
college reflect the second principle of the ATD culture of evidence model: use of data to 
improve student success. 

IT Capacity 

All of the Washington colleges initially had trouble accessing the data needed to 
conduct the longitudinal cohort tracking recommended by ATD because they were dependent 
on an antiquated statewide “legacy” student information system. At least two of the colleges — 
WA-B and WA-D — built data warehouses to facilitate longitudinal analysis of data, although 
others, including WA-E and WA-F, downloaded data from the state system to conduct their 
analyses. 

IR Capacity and Effectiveness 

Three of the colleges — WA-B, WA-C, and WA-D — added IR or IT staff under 
ATD. One (WA-D) built an institutional effectiveness office from scratch (see the case study 
below, Box 2.1, Building an Effective Home-Grown Institutional Effectiveness Function). 
However, the increased IR capacity at these colleges, combined with the greater attention to 
data and research, also increased internal demand for information. As a result, IR staff reported 
that they were still spread thin. Both WA-E and WA-F lost their lone IR staff person in the first 
or second year of the initiative. In both cases, it took time to find a replacement, and they were 
thus slow in building their IR capacity. All of the colleges reported that they could use 
additional research staff to meet the demand for research for compliance reporting and college 
promotion as well as to support improvements in programs and services. 
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Four of the colleges — WA-A, WA-B, WA-C, and WA-D — supplemented their IR 
capacity with data teams that helped filter requests for information, determine priorities for 
research, and build awareness among their colleagues about the potential benefits of collecting 
data and using it effectively. The IR directors at two of the colleges (WA-A and WA-C) trained 
colleagues to use data analysis software. At WA-A, deans, department chairs, faculty, and 
others began to conduct their own analyses to address their information needs (see the case 
study below, Box 2.2, Democratizing Data Use). They brought questions they could not answer 
to data team members, who worked with the IR director to find answers. These four colleges 
used such approaches and the resourcefulness of their IR staffs to compensate for their lack of 
trained IR personnel and to build IR functions viewed as responsive by their colleagues. In the 
two remaining cases (WA-E and WA-F), the IR directors worked mainly for the president and 
senior leadership and focused on compliance reporting, leaving little time to work with faculty 
and staff. 
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Box 2.1 


Case Study: Building an Effective Home-Grown 
Institutional Effectiveness Function 

For a variety of reasons related to institutional capacity and faculty and staff inexperience 
with effective date use, WA-D had attempted to establish a research office more than once prior to 
joining Achieving the Dream, without success. 

To increase the possibility of success, the vice president of instruction identified some of 
the “power users” at the college and gave them release time to work with the newly formed Office of 
Institutional Effectiveness (OIE). The vice president selected a faculty member to coordinate the 
OIE who had credibility among her peers because she had taught courses in all three instructional 
divisions; had headed the faculty union; and had been a grant writer, which made her familiar with 
data and the other power users at the college. She took the power users on a retreat to discuss how 
they would create more systematic processes for collecting, analyzing, and reporting data. The OIE 
focused on identifying and understanding the problems the power users had experienced with data so 
they could help others in the campus community solve their own problems and also send the 
message that data would be used to support their work, not to evaluate it. 

The OIE built a data warehouse for use in tracking cohorts of students over time and began 
leading studies to inform efforts to improve programs and services at the college. For example, the 
college used student and staff focus group data to determine the need for a revised orientation and 
advising process. Data analysis also revealed the need to address developmental math because 85 
percent of incoming students were placing into at least one level of developmental math. A survey of 
students developed by the college (similar to the Community College Survey of Student 
Engagement) revealed that students wanted the college to be more prescriptive, which led to the 
revised orientation and advising process. In another case, OIE staff worked with the psychology 
department to analyze placement test scores and pass rates for students in their courses to determine 
the need for prerequisites. Although there was initially some concern about the impact on course 
enrollments, the department implemented the prerequisites and saw increases in course completion. 
As word of the psychology department changes spread, the other social sciences departments began 
asking for data. 

The college also used analyses conducted by the OIE to decide which programs and services 
to keep and which to cut. A math and study skills learning community strategy implemented under 
ATD was evaluated, and the evaluation found that students receiving this treatment did not have 
better outcomes than students in stand-alone math courses. This intervention was also costly, so it 
was discontinued, and the math department developed a lengthened, three-quarter algebra sequence. 
In another case, the college decided not to cut the writing center, math lab, or learning centers 
because data showed that these were important to student success. Instead, the college made them 
self-supporting by charging students a fee that covered the costs of these centers. 


16 




Box 2.2 


Case Study: Democratizing Data Use 

WA-A began its work with Achieving the Dream during a period when the college 
was transitioning between institutional research directors. Because there was no obvious 
leadership for the ATD data team (a position frequently filled by a college’s IR director and 
staff), the college asked faculty and staff who were not necessarily data-savvy to participate 
on the team. This approach to building the data team had several positive consequences. 
First, it connected the team to a larger range of stakeholders at the college. Faculty and staff 
who participated on the data team were able to share what they learned about data collection 
and analysis with colleagues during department meetings, professional development 
workshops, and informal interactions. This fostered a sharing of approaches to using data 
that the college’s IR director referred to as the “democratization of data” because it helped to 
involve faculty and staff who were previously unaccustomed to using data and research to 
improve practice. The WA-A case shows that Achieving the Dream principles can be 
applied in a college with a culture of bottom-up decision making. 


Use of Data for Identifying Gaps and Formulating and Evaluating Strategies 

Four of the colleges — WA-A, WA-B, WA-C, and WA-D — used longitudinal cohort 
analysis at least to some extent to identify the target populations for their ATD strategies. The 
Achieving the Dream approach of examining the progress and outcomes over time of student 
cohorts grouped by their characteristics was new to many of those we interviewed at the 
Washington colleges. For example, administrators at WA-C said that faculty and staff at their 
college were slow to understand and accept data on student achievement gaps. One reason was 
the perception among some faculty that the faculty was being blamed for poor student 
performance. Administrators said that it took at least two years before the high level of mistrust 
began to subside. Administrators we interviewed attributed the positive change to persistent 
efforts in multiple venues and occasions to present data on the problem and promote the idea 
that addressing it was a campus-wide responsibility, not just one the faculty must own. 

Only WA-A and WA-D systematically used data to formulate student success 
strategies. At WA-D, staff from the office of institutional effectiveness worked with faculty and 
administrators to design intervention strategies and/or grants intended to address achievement 
gaps, providing longitudinal data, helping to shape strategies and objectives, and developing 
evaluation plans. The math faculty at WA-A were convinced by data from student focus groups 
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and surveys to move from an individual mentoring approach to a larger department-wide 
curricular redesign in order to address the needs of the large numbers of developmental math 
students more efficiently. Before ATD, discussions about achievement gaps at the college were 
abstract or based on anecdotal evidence. ATD helped the college pinpoint where those gaps 
existed and identify at-risk groups (such as the ESL new-immigrant population). The college 
used outside evaluators and internal evaluations (based on focus groups, surveys, and 
participation analyses) to understand the effects of programs and services on student success. 
Even so, college leaders acknowledged that the college did not have a systematic approach to 
evaluating its efforts to improve student success and to using the results to make further 
improvements. In contrast, the IR director at WA-B worked with colleagues to build in 
evaluation as strategies were being designed, and he encouraged colleagues who were writing 
grants to be explicit about how they would measure success. In the other four colleges, 
evaluation was an area still in need of improvement. 

Broad Engagement in the Improvement Process 

Table 2.2c summarizes our assessment of the extent to which the practices of each 
college reflect the third principle of the ATD culture of evidence model: broad engagement in 
the improvement process. 

Faculty and Student Services Staff 

Engagement of faculty and student services staff in using data and research to improve 
student success is one of the key features distinguishing colleges that were further along in 
building a culture of evidence from those where such a culture was not well developed. 
Decision making at WA-A, the college that was furthest along in creating a culture of evidence, 
was described by those we interviewed as “non-hierarchical” and “bottom-up.” The fact that the 
student success strategies implemented at this college were the product of all-department 
efforts, not small groups of entrepreneurs, reflects the general culture of collegiality and 
cooperation at the college. Due in part to the influence of ATD, discussions of data became a 
regular part of department meetings and strategic planning sessions, and faculty and staff used 
them to decide how best to serve students. A small percentage of part-time faculty (roughly 10 
percent) participated in professional development related to student success. 

Although faculty engagement in evidence-based decision making remained very limited 
at two of the colleges (WA-E and WA-F), the remaining three colleges made clear progress in 
the first five years they participated in ATD. After some initial resistance, more academic 
departments at WA-C began using data as part of departmental discussions. The decision by 
WA-D to choose a well-regarded faculty member to head the newly established office of 
institutional effectiveness sent the message to the faculty that research would focus on 
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instruction (see the case study above, Box 2.1, Building an Effective Home-Grown Institutional 
Effectiveness Function). WA-D also encouraged greater engagement with data at all-campus 
convocations and professional development sessions designed to encourage faculty to use IR 
services. At WA-B, institutional research was generally seen as irrelevant to the academic 
departments, but faculty began to see its importance. In one case, faculty in a program slated to 
be eliminated due to budget cuts worked with the IR staff to get placement data and convinced 
the board that the program was still viable and should be retained. 

Faculty engagement is, not surprisingly, particularly crucial for efforts to improve 
instruction. As described in more detail in the next chapter, faculty at four of the colleges were 
instrumental in developing the instructional innovations implemented as part of ATD. Some 
faculty at WA-B objected to what they perceived as the top-down way in which strategies were 
initially developed under ATD. A group of faculty was successful in introducing an innovative 
approach to teaching reading, which was adopted by instructors from across academic 
departments. Although they got the idea from a session at one of the ATD Strategy Institutes, 
the faculty who led this effort explicitly contrasted the grassroots manner in which they 
introduced and spread this innovation to what they perceived as the top-down orientation of 
ATD. The vice president for instruction admitted that the initial approach to developing 
strategies under ATD was misguided, acknowledging that the most successful instructional 
reforms at the college have all had strong instructor buy-in. According to this vice president, 
“Instead of saying ‘Okay, you’re going to be part of this initiative,’ faculty need to say ‘I want 
to be part of this effort.’” 

Letting faculty take the lead does not always work, however. At WA-D, administrators 
were determined to let the math faculty decide what to do and to experiment with different 
approaches. However, faculty within the department disagreed about the best approaches, and 
as a result the efforts to improve outcomes were disconnected and generally small in scale. 
Administrators at WA-C seemed to have found a better balance between prodding faculty to 
improve instruction and giving them the flexibility to decide how best to do so. Particularly in 
the last couple of years, faculty have taken the lead in introducing innovations, such as 
University Design for Learning (UDL), discussed in the next chapter. At WA-F, a group of 
entrepreneurial faculty (including some adjuncts) collaborated to create integrated assignments 
that encouraged students to think in an interdisciplinary manner. However, because of the lack 
of strong support from the college for these efforts and for helping to spread instructional 
innovations across departments, this innovation was confined to a relatively small number of 
faculty and students. 
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Instruction-Student Services Collaboration 


At two of the colleges, collaboration between student services and academic affairs was 
already strong even before the colleges joined ATD. Although WA-A’s student services staff 
were not heavily involved in the development of the ATD strategies, which were primarily 
focused on improving academic programs, both student services staff and academic faculty and 
administrators reported that collaboration was strong. At WA-B, most of the professional 
advisors were “embedded” in particular program areas. Faculty and student services staff 
reported that this strategy facilitated strong collaboration with one another. At WA-F some 
advisors were also assigned to specific divisions although, overall, the extent to which student 
services staff worked with faculty to improve programs and practices was less than at WA-B. 
At WA-C, the work by ATD core and data teams to plan and oversee the implementation of 
strategies created more opportunities for collaboration between faculty and student services 
staff, although our interviewees indicated that there was still room for improvement. Student 
services and academic affairs personnel at WA-D and WA-E reported that although they 
sometimes worked together, they often worked in their respective “silos.” This increased 
opportunities for collaboration somewhat, although the extent to which student services staff 
worked with faculty to improve programs and practices was greater at WA-B and at WA-F. 

Students 

Some of the colleges regularly conducted surveys and focus groups with students to 
better understand their needs. For example, WA-D had success with a survey instrument that 
was similar to the Community College Survey of Student Engagement (which is designed to 
measure student engagement by assessing student behaviors and institutional practices) but less 
expensive and customized to the college’s particular information needs. However, there was 
little evidence of an established approach for involving students actively in efforts to improve 
student success at any of the colleges. 

External Stakeholders 

Of the colleges, WA-C made the most concerted effort to involve external stakeholders 
in improving student success. The college worked with numerous feeder high schools to 
conduct ACCUPLACER testing during students’ eleventh and twelfth grades and worked with 
high school counselors on matriculation agreement updates and course selections. The college 
was an active member of a local network involving partners from K-12, a local foundation, 
other local colleges and universities, and the local housing authority. These relationships 
allowed the college’s students to access the support services offered by other network partners. 
One board member of the college was a program officer at a local foundation that helped recruit 
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more faculty of color at the college and educated faculty on how to better address cultural needs 
and interests of students of color in the classroom and on campus. 

WA-A began to focus on outreach to community groups and building partnerships with 
organizations in the surrounding area. Its president worked to involve local chambers of 
commerce, economic development groups, poverty-reduction programs, and local businesses in 
ATD efforts and other important conversations. In addition, the college held several events to 
engage the rapidly growing local communities of color with the college culture, including a 
day-long, Spanish-language-only event for Latinos in the community. Planning is also 
underway for a youth summit based on the model of the “Black, Brown, and College-Bound” 
event that is held annually at Hillsborough Community College in Tampa, Florida — one of the 
Round 1 colleges. These efforts resulted from a heightened awareness of the changing 
demographics at the college. 

While the other four colleges had partnerships with local high schools and community 
groups, none seemed to be capitalizing on these relationships to support and leverage efforts 
within the college to improve student outcomes. 

Continuous Improvement 

Table 2.2d shows our assessment of the extent to which the colleges built an 
infrastructure to support continuous improvement in accordance with the fourth principle of the 
ATD culture of evidence model. 

Strategic Planning 

As reported in Chapter 1, the Northwest Commission on Colleges and Universities 
recently revised its standards to place more emphasis on the use of evidence in planning and 
decision making in the institutions it accredits. A senior administrator at WA-A said that, as a 
result of a confluence of forces — including increased demand for evidence for accreditation; 
the Student Achievement Initiative (a statewide performance funding initiative); and, most of 
all, the economy and the state budget cuts — “programs without evidence of effectiveness will 
be on the chopping block.” 

The Washington ATD colleges are all responding to these new pressures for evidence- 
based decision making, albeit in somewhat different ways. For example, WA-B formalized and 
strengthened its program review process to include a focus on student retention and completion, 
not just enrollment. The college used this process to decide which programs to eliminate in 
response to sharp cuts in state funding. The administration worked with the faculty to develop 
the metrics to be used. Faculty and department chairs interviewed said that the administration 
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was trying to be more transparent and fair in its decisions. The college’s strengthened IR 
capacity and program review process and measures helped to make this possible. 

In response to the new accreditation standards, all of the Washington colleges 
developed or were in the process of creating “core themes” or goals for student success with 
measureable indicators. Two of them — WA-C and WA-D — said that they borrowed from the 
ATD framework and metrics to define their core themes and measures related to student 
success. The others said they were influenced by Achieving the Dream and other performance 
measurement frameworks, including the “achievement point” metrics associated with the 
Student Achievement Initiative, Washington State Board for Community and Technical 
College’s performance funding policy. 4 Although they were guided by ATD and other 
frameworks, each college developed its own set of metrics based on its particular goals and 
internal process for defining goals and measures. It should be noted that none of the Washington 
colleges devoted much time to examining their performance in relation to the ATD performance 
metrics (which are described in more detail in Chapter 5). We will return to this point in the 
recommendations presented in the final chapter of this report. 

Organizational Infrastructure 

Four of the colleges transformed their ATD core and data teams into standing 
committees responsible for overseeing efforts to improve student outcomes. WA-C merged its 
ATD core and data teams into an institutional effectiveness leadership team with responsibility 
for overseeing student success initiatives, deciding which strategies to continue, and improving 
collaboration between academics and student services. This body advised the president and the 
board. Responding to the new accreditation requirements, WA-B converted its ATD core and 
data teams into standing strategic planning and institutional effectiveness teams that met once a 
month and advised the executive cabinet. 

Because the administration at WA-D did not want to create a separate, parallel set of 
committees for its ATD work, the college integrated its core and data teams into the main 
governance structure of the college. The core team became the institutional effectiveness (IE) 
team, and the data team became the Office of Institutional Effectiveness (OIE). The IE team 
was the central body on campus focused on student success, charged with guiding reform 
efforts based on data and working with the OIE on the accreditation process. Helping to 
determine the research priorities of the college and define the indicators for measuring progress 
on the core themes of the college, it met two to three times each year and was broadly 
representative of faculty and staff. It also worked with the OIE to identify opportunities for 
improvement and evaluate improvement strategies (see the case study above, Box 2.1, Building 
an Effective Home-Grown Institutional Effectiveness Function). 

4 For more information about the Student Achievement Initiative, see Shulock and Jenkins (201 1). 
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WA-A already had a Student Outcomes and Competencies Committee to oversee 
student success efforts and the setting of learning outcomes. Responding to the push from the 
college’s ATD coaches to use evidence for decision making, the committee made reporting 
requirements much more rigorous. It worked with the data team established under ATD to 
expand access to and use of data on students throughout the campus. It also showed the value of 
creating a central body to coordinate student support efforts and provide access to data and 
analysis. 

The remaining two colleges have yet to establish a committee to monitor and guide 
efforts on campus to improve student success. WA-E had an instructional council that advised 
the college leadership on issues related to academic programs, but its focus was primarily on 
program management and regulation, not on improving student outcomes. The college was 
considering establishing a body that would be responsible for overseeing student success 
initiatives but had not done so at the time of our research. WA-F had a strategic enrollment 
management committee that focused on recmitment but nobody to oversee efforts to improve 
student retention and completion. 

Professional Development 

Faculty development at the colleges tended to be decentralized, varying by department, 
and based on the interest of individual faculty members — in part as a result of lack of 
resources. For example, WA-B eliminated a full-time faculty development staff person whom 
faculty interviewees praised for helping to support efforts to improve instruction. The 
alternative, according to the college’s administrators, was to reduce resources for instruction. 
The college used ATD funds to enable a small group of faculty to travel for training in the 
Reading Apprenticeship model developed by WestEd, which they subsequently helped to 
spread across academic departments. 

Three of the colleges used ATD funds for professional development. WA-D provided 
training related to better use of data, and WA-E and WA-C provided professional 
developmental for those involved in specific ATD strategies. Only WA-A had a process for 
supporting faculty development in a way that was strategically tied to its student success efforts. 
A report from the college to ATD on its activities in 2010-201 1 indicated that while 80 percent 
of faculty participated in professional development that reinforced student success efforts, a 
large majority of student services staff (roughly 95 percent) did not participate. 
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Achieving the Dream Colleges in Washington State 


Table 2.2a 

Implementation of Culture of Evidence Principle 1, Committed Leadership, at Washington ATD Colleges, Spring 2011 


Indicator 

WA-A 

WA-B 

WA-C 

WA-D 

WA-E 

WA-F 

President and senior leadership 

Very high 

High 

Very high 

High 

Low 

Low 

Board leadership 

Medium 

Medium 

Medium 

Medium 

Medium 

Medium 

Faculty leadership 

High 

High 

Medium 

Medium 

Low 

Low 

Commitment to closing achievement 
gaps 

Very high 

High 

High 

High 

Low 

Very low 

Summary Score 

High 

High 

High 

Medium/High 

Low 

Low 


Table 2.2b 

Implementation of Culture of Evidence Principle 2, Use of Evidence for Improvement, at Washington ATD Colleges, 

Spring 2011 


Indicator 

WA-A 

WA-B 

WA-C 

WA-D 

WA-E 

WA-F 

IT capacity 

Medium 

Medium 

Medium 

Medium 

Low 

Low 

IR capacity 

Medium 

Medium 

Medium 

Medium 

Low 

Low 

IR effectiveness 

High 

High 

High 

High 

Low 

Very low 

Use of data for identifying gaps 

High 

High 

High 

High 

Low 

Low 

Use of data for formulating strategies 

High 

Medium 

Medium 

High 

Very low 

Very low 

Evaluation of strategies/use of 
findings 

High 

High 

Medium 

Medium 

Low 

Low 

Summary Score 

High 

Medium/High 

Medium 

Medium/High 

Low 

Low 
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Achieving the Dream Colleges in Washington State 


Table 2.2c 


Implementation of Culture of Evidence Principle 3, Broad Engagement, at Washington ATD Colleges, Spring 2011 


Indicator 

WA-A 

WA-B 

WA-C 

WA-D 

WA-E 

WA-F 

Full-time faculty 

Very high 

High 

Medium 

High 

Low 

Very low 

Part-time faculty 

Medium 

Low 

Low 

Low 

Very low 

Very low 

Student services staff 

High 

High 

High 

Medium 

Medium 

Low 

Academic-student services 
collaboration 

High 

Medium 

Medium 

Medium 

Low 

Medium 

Students 

Low 

Low 

Low 

Low 

Low 

Low 

External stakeholders 

Medium 

Medium 

High 

Medium 

Low 

Low 

Summary Score 

Medium/High 

Medium 

Medium 

Medium 

Low 

Low 


Table 2.2d 


Implementation of Culture of Evidence Principle 4, Continuous Improvement, at Washington ATD Colleges, 

Spring 2011 


Indicator 

WA-A 

WA-B 

WA-C 

WA-D 

WA-E 

WA-F 

Strategic planning 

High 

High 

High 

High 

Low 

Low 

Organizational infrastructure 

High 

High 

High 

High 

Low 

Very low 

Professional development 

High 

Medium 

Medium 

Medium 

Low 

Low 

Summary Score 

High 

High 

High 

High 

Low 

Low 
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Summary of Changes in Organizational Practice Because of ATD 

Some of the practices described above were already in place when the Washington 
colleges joined ATD. Table 2.3 summarizes the major changes in organizational practice 
undertaken by each college as a direct result of their participation over five years in ATD. 


Achieving the Dream Colleges in Washington State 

Table 2.3 

Major Changes in Organizational Practice Among the Washington Colleges 



During the First Five Years of ATD 

College 

Changes in Organizational Practice 

WA-A 

• Developed “core theme” goals and measures for improving student success. 

• Set measurable goals for improving student outcomes in all departments. 

• Created data team to expand access to data and research to more faculty and staff. 

• Developed process whereby academic and student services staff work together to 
use data to identify groups falling behind and devise sttategies for improving 
student outcomes. 

WA-B 

• Developed “core theme” goals and measures for improving student success. 

• Hired 1R director and analyst who built data warehouse and strengthened data 
analysis and evaluation. 

• Evolved ATD core and data teams into standing strategic planning and institutional 
effectiveness committees. 

• Strengthened program review process. 

WA-C 

• Developed “core theme” goals and measures for improving student success. 

• Added IR staff and increased information systems capacity. 

• Strengthened and systematized approach to developing and implementing student 
success strategies. 

• Merged ATD core and data teams to create standing IE leadership team. 

WA-D 

• Developed “core theme” goals and measures for improving student success. 

• Created standing institutional effectiveness team (IE team) and IE office (OIE). 

• Created data warehouse and hired programmer. 

• Established procedures for making requests for information from research staff. 

WA-E 

• Incorporated student success goals into college academic master plan. 

• Increased analysis by IR for use by president, senior executives, and grant writing. 

WA-F 

• Developed “core theme” goals and measures for improving student success. 
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All of the colleges developed goals for student success with measures that were similar 
to and in some cases adapted from the ATD performance measures. WA-A went a step further 
by asking each department to establish student success goals and measures tied to the 
institutional goals. WA-B strengthened its program review process, with performance measures 
defined in collaboration with faculty. 

Three of the colleges — WA-B, WA-C, and WA-D — added institutional research staff 
who built data warehouses, improved procedures for requesting information, provided data 
support to student success committees and others, and trained faculty and staff in the use of 
data. At these three colleges and at WA-A, the efforts of the IR staff to increase access to data 
and research were enhanced by “data teams” comprised of faculty, staff, and administrators who 
not only helped to promote the use of data to their colleagues but also provided guidance on 
how to do so. 

The three colleges that added IR staff (WA-B, WA-C, and WA-D) also created 
standing bodies that were responsible for overseeing efforts to improve student outcomes. Such 
a committee already existed at WA-A prior to its joining ATD. In some cases the data teams 
were integrated into these committees; in others, the data teams were separate. In all cases, IR 
staff worked closely with committee members to help ensure that the decisions they made were 
based on evidence. In establishing these committees, these colleges also created or strengthened 
processes for identifying opportunities to improve student success and developing strategies for 
doing so. 


Comparison With the Round 1 Achieving the Dream Colleges 

As noted, in funding an evaluation of the ATD work in Washington State, Lumina 
Foundation and other funders were particularly interested in seeing whether the Washington 
colleges, having joined the initiative two years after it began (i.e., in the third round), would 
make faster progress in adopting the ATD culture of evidence principles than colleges that 
joined in the first round. (The experiences of the Round 1 colleges during their first five years 
with the initiative were examined in a 2011 report by MDRC and CCRC [Rutschow et al., 
2011 ]). 


While it is difficult to compare the six Washington colleges with the 26 Round 1 
institutions, it does seem that the Washington colleges, as a group, went at least as far as did the 
Round 1 colleges in building a culture of evidence. 

Figure 2.1 shows the extent to which the six Washington colleges had adopted the ATD 
culture of evidence principles between 2008, when we conducted our first round of field 
research, and spring 2011, when we made follow-up visits. Figure 2.2 shows the changes in 
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level of adoption among the 26 Round 1 colleges between 2006 and 2009. All but one of the 
Washington colleges made progress in implementing the ATD culture of evidence principles. 
Two of them, which early on in the initiative had only begun to implement practices consistent 
with the ATD principles, made very substantial progress later on. Five years after they joined 
ATD, we rated four of the six Washington colleges as having implemented the ATD principles 
at a high level. In comparison, only 1 1 of the 26 Round 1 colleges (42 percent) implemented 
practices associated with a strong culture of evidence by the end of their five-year ATD 
implementation period. 


Achieving the Dream Colleges in Washington State 
Figure 2.1 

Culture of Evidence Implementation: Washington ATD Colleges: 
Spring 2008 and Spring 2011 


Spring 2008 



Low, 2 


Spring 2011 



28 


Achieving the Dream Colleges in Washington State 


Figure 2.2 

Culture of Evidence Implementation: Round 1 ATD Colleges, 
Spring 2006 and Spring 2009 


Spring 2006 


Spring 2009 
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Chapter 3 

Student Success Strategy Implementation 


This chapter examines strategies for improving student success that the six Washington 
colleges were implementing in spring 2011. It describes the colleges’ various strategies and 
compares them with the strategies implemented by the first group of colleges to join the 
initiative in 2004—2005 (the Round 1 colleges) that were examined in a 201 1 report by MDRC 
and CCRC (Rutschow et al., 2011). 

For our review of Washington ATD colleges’ strategies, we classified specific 
interventions according to their size: small, medium, or large. Small-scale strategies are those 
that reached 10 percent or less of their targeted population, medium-scale strategies reached 
between 10 percent and 25 percent of their targeted population, and large-scale strategies 
reached more than 25 percent of their targeted population. 5 Similarly, we used three 
classifications of intensity: low, medium, or high. Low-intensity strategies reached students for 
five or fewer hours per term, medium-intensity strategies reached students for five to ten hours 
per term, and high-intensity strategies reached students for more than ten hours per term. 

The 2011 report by MDRC and CCRC found that most of the strategies implemented 
by the Round 1 colleges were either small in scale and/or relatively low intensity. Further, most 
involved changes to student services and academic support programs rather than enhancements 
of classroom instruction. The report concluded that ATD colleges were unlikely to improve 
outcomes for a substantial number of students unless they were able to implement their 
strategies on a larger scale and with greater intensity. Building on the findings of that report, and 
drawing on research on high-performing educational institutions and non-education 
organizations and their implications for community colleges, this study focused on the process 
used by the Washington colleges that were successful in implementing student success 
strategies at scale. This chapter reports on those efforts and also on work by faculty at the 
Washington colleges to develop new approaches to instruction and to support the adoption of 
instructional innovations by their colleagues. 

The following are the main findings from our examination of student success strategies 
in the Washington colleges: 

• Five of the six colleges implemented at least one large-scale strategy. 


5 These categories of scale, developed for the report on the Round 1 colleges (Rutschow et al., 201 1), 
made comparisons of scale possible across the two groups of colleges. We asked ATD project staff at 
each college to estimate a scale range for each of their strategies. 
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The colleges focused their strategies more on improving instruction and 
developed more strategies at a large scale than did the Round 1 colleges. 


• Several colleges sought to engage faculty in the development and diffusion 
of instructional reforms, with varying degrees of success. 

• The colleges that went furthest to implement innovations at scale by 
redesigning programs and services were those that were further along in 
developing a culture of evidence. 

The majority of student success strategies across the Washington colleges targeted 
students in developmental education and first-year students. While the strategies aim at 
improving college readiness and persistence among entering students, as discussed in the 
concluding chapter of this report, colleges may need to focus more attention on reforming 
college-level programs and supports to ensure that students complete their programs and are 
prepared for further education and to ensure that students in career-technical programs are 
prepared for advancement in the labor market. 


Student Success Strategies of the Washington Colleges 

In spring 2011, the six Washington colleges were implementing nearly 30 “direct” 
student interventions that they developed as part of Achieving the Dream. Direct strategies are 
those that reach students personally (as opposed to, say, faculty development, that affects 
students indirectly through faculty). 6 Table 3.1 summarizes the main strategies implemented in 
instruction and student services by the six colleges. It indicates that many of the changes to 
instruction occurred in developmental education programming and that many changes to 
student services were designed to benefit incoming and first-year students. WA-B and WA-F 
confined their ATD strategy development efforts to innovations in instruction. 


6 We limit our discussion in this chapter to strategies that affect students directly. Chapter 2 covers 
other ATD activities, such as increasing institutional research capacity and strengthening faculty 
professional development. This distinction between direct student strategies and other ATD activities was 
also made in the evaluation of the Round 1 college strategies (Rutschow et al., 201 1). 
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Achieving the Dream Colleges in Washington State 


Table 3.1 


Main Changes in Instructional Programs and Student Services at the 
Washington Colleges Under ATD 


College 

Instruction 

Student Services 

WA-A 

• Developmental math sequence 
transformed to make it relevant to 

• ESL transition center — financial aid 
and other assistance to help basic 


students’ academic paths. 

skills students transition to college 


• Revised developmental English. 

level coursework. 

WA-B 

• Reading Apprenticeship — discipline- 
based literacy instruction — 

• None. 


instituted. 



• New math tutoring center. 


WA-C 

• Universal Design for Learning 
instituted. 

• Advising for first-year students 
revamped. 


• Supplemental instruction in math and 

• Standardized student success seminar 


English provided. 

improved. 



• Advisor dashboard documenting 
student satisfaction with services 



instituted. 

WA-D 

• Developmental math restructuring 
(still in process). 

• New student orientation and advising 
revised. 

WA-E 

• Pre-college math courses restructured. 

• New student orientation revised. 



• Student success course created. 



• Mentoring program for Latino 
students added. 

WA-F 

• Cohorts and study sessions for 

• None. 


developmental math sequence created. 



• Integrative learning assignments 
implemented. 



NOTE: This table presents the main ATD strategies in instruction and student services at the six colleges 
but is not an exhaustive list of the direct strategies implemented as part of ATD. 


Figure 3.1 shows the distribution of direct student strategies by category across the six 
colleges. Changes in classroom instruction, which include curriculum and instruction and 
learning communities, comprise over a third of the direct strategies. The number of strategies 
that each college implemented varied significantly, ranging from only two to a high of eight. 
Three of the six Washington colleges — WA-A, WA-B, and WA-D — had implemented only 
two or three direct strategies at the time of this study. 
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Achieving the Dream Colleges in Washington State 
Figure 3.1 

Distribution of ATD Direct Strategies Implemented at the Washington Colleges, 

Spring 2011 

Assessment 

1 \ 


Advising and 
Mentoring 
8 


Academic 

Support 

8 


Comparison Between the Washington and Round 1 Colleges 

The spring 2011 report by MDRC and CCRC on the progress of the 26 Round 1 
colleges had three main findings regarding the implementation of strategies for improving 
student success (Rutschow et al., 2011). First, only one quarter of all direct student interventions 
in the Round 1 colleges were designed to change classroom instruction; most of the 
interventions involved either student support services or academic supports that were designed 
to improve instruction in the classroom. Second, there was a trade-off between the scale and 
intensity of implemented ATD strategies across the Round 1 colleges. Low-intensity strategies, 
which tended to be support services such as orientation and advising programs, were more 
likely to be implemented at scale than were high-intensity strategies such as curricular reforms, 
supplemental instruction, and learning communities, which generally reached relatively small 
numbers of students. Third, while some of the colleges did implement strategies that touched 
large numbers of students, the majority of all direct interventions at the Round 1 colleges 
remained small in scale. These findings show the difficulty of implementing interventions on a 
scale sufficient to reach substantial numbers of students, especially high-intensity instructional 
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reforms. Yet, as MDRC and CCRC pointed out in the 2011 report, unless colleges can 
implement high intensity instructional reforms at scale, they are unlikely to be able to increase 
the completion and success rates for substantial numbers of students. 

The Washington State colleges and the Round 1 colleges focused on similar 
populations for the majority of direct strategies. Both groups of colleges targeted developmental 
education for the majority of their interventions. At the Round 1 colleges, nearly half of all 
direct strategies focused on this area, and at the Washington colleges the proportion was even 
higher (56 percent). Furthermore, in both groups of colleges, strategies targeting first-year 
students were the second most common after developmental education. 

One notable difference between the Round 1 and Washington colleges was that the latter 
focused their ATD activities on instruction to a greater extent than did the Round 1 colleges. 
Among the Round 1 colleges, instructional changes (including both reforms to curriculum and 
instruction and instructional supports) comprised 52 percent of all direct student success 
strategies. Among the Washington colleges, instructional changes accounted for 70 percent of 
all direct strategies (see Figure 3.2). Therefore, the Washington colleges may have benefited 
from the experiences of the Round 1 colleges and learned about successful approaches to 
change instmctional practices. It is also noteworthy that the Round 1 colleges implemented a 
wider variety of student services strategies than did the Washington colleges. For example, 
some Round 1 colleges implemented early alert systems and scholarship and financial aid 
programs, two types of interventions that were not implemented at any of the Washington 
colleges. 
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Achieving the Dream Colleges in Washington State 
Figure 3.2 

Proportion of Student Success Strategies by Type Used at 
the Round 1 and Washington ATD Colleges 

Round 1 Washington 


Curriculum 
and Instruction 
27 % 

Instructional 

Supports 

25 % 


Student 

Support 

Services 

48 % 


Curriculum 

and 

Instruction 

41 % 


Student 

Support 

Services 

30 % 


Instructional 

Supports 

29 % 


Another difference between the Round 1 and Washington colleges was the scale of 
student interventions. Institution-wide improvements to student outcomes require interventions 
that reach a substantial number of students. Thus, Achieving the Dream encourages colleges to 
develop strategies with the potential to benefit students on a large scale. While the Washington 
colleges had varying degrees of success developing large-scale strategies in student services and 
instruction, five of the six colleges implemented at least one large-scale strategy. 

Neither the Round 1 college group nor the Washington State college group was able to 
bring more than half of its strategies to over 25 percent of their target populations, although the 
Washington colleges did have a greater percentage of large-scale strategies than the Round 1 
colleges after four years. Figure 3.3 shows that 31 percent of strategies were operating at scale 
at the Round 1 colleges, compared with 43 percent of strategies at the Washington colleges. 
Thus, the Washington colleges may have learned from the experience of the Round 1 colleges, 
many of which early on tried to implement too many interventions that were not well connected 
with one another (Rutschow et al., 2011). Three of the Washington colleges — WA-A, WA-B, 
and WA-C — initially developed a number of small, boutique strategies but soon shifted to 
strategies that would have a more systemic impact. For example, one college initially planned to 
implement more than 20 strategies but was encouraged by its ATD coaches to focus on 
interventions that had a broader impact on students. 
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In contrast with the Round 1 colleges, the large-scale strategies at the Washington 
colleges were not predominantly aimed at student services. Over half the strategies that had 
reached a substantial number of students in Washington involved changes to instructional 
practices. It is a potentially positive sign that among the Washington colleges more strategies 
were being developed at scale and more of them involved instructional reforms than in the 
Round 1 colleges. It is also notable that at the Washington college that had gone furthest to 
implement the ATD culture of evidence principles — WA-A — all three of its interventions 
were considered to be large scale (see Chapter 2). 


Achieving the Dream Colleges in Washington State 
Figure 3.3 

Proportion of Strategies Operating at Small, Medium, or Large Scale at 
the Round 1 and Washington ATD Colleges 

Round 1 Washington 


Large 
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The Redesign of Programs and Support Services at Scale 

The research literature on organizational effectiveness suggests that no one innovative 
practice or even set of discrete practices is sufficient to improve organizational performance. 7 
The implication for community colleges is that in order to achieve substantial improvements in 


7 For a review of research on organizational effectiveness and improvement in and outside education, 
and the implications for community colleges, see Jenkins (201 1). 
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student outcomes they will need to make systemic changes in programs and services: no single 
innovation or even set of interventions will be sufficient to dramatically increase the number of 
students who complete their education. From this perspective, rather than test and scale up 
discrete interventions, colleges need to redesign major programs and functions at scale. The 
research on organizational effectiveness makes clear that redesign efforts need to be well 
aligned to achieve organizational goals — which in the case of ATD colleges is improved rates 
of student completion. 

Another lesson from research on effective organizations is that for innovations to 
diffuse through a college and be sustained over time, faculty and staff must be broadly engaged 
in the innovation process. Faculty in particular must be empowered and supported to lead 
efforts to improve instruction. The report on the Round 1 colleges indicated that many of them 
had difficulty engaging faculty. Chapter 2, which discusses organizational changes, indicates 
that faculty engagement was also a challenge for some of the Washington colleges. 

Some of the Washington colleges did attempt to make systemic changes to programs 
and services. And at some of the colleges a substantial number of faculty members were 
experimenting with ways to improve instruction, and in some cases their colleges supported 
them in efforts to disseminate the innovations throughout their campuses. These efforts are 
described below. 

Reforms to Programs and Services 

As noted, the majority of the direct interventions implemented as part of ATD in 
Washington were focused on reforms to instruction. Instructional strategies included changes to 
curriculum and instruction and instructional supports such as tutoring, supplemental in-class 
instruction, and other programs to support classroom learning. Student services strategies 
included changes to orientation and advising and student success courses. Below we describe 
several notable efforts to redesign programs and processes in student support services and 
instruction. 

Student Services 

Much of the reform work in student services involved redesigning programs or creating 
new interventions for students at the beginning of their educational experience. At four of the 
colleges — WA-A, WA-C, WA-D, and WA-E — a student services intervention was one of the 
major efforts conducted as part of ATD. Three colleges (WA-C, WA-D, and WA-E) revised the 
intake and advising process for incoming students. The fourth college (WA-A) developed a 
transition center for ESL students. 


38 



One Washington college — WA-C — developed a comprehensive approach to helping 
first-year students navigate from matriculation to the declaration of a program of study, with the 
goal of helping them obtain 30 credits by the end of the first year. The multipronged approach 
consisted of practices that existed prior to Achieving the Dream and new strategies developed as 
part of the initiative; they included early assessment, mandatory orientation, a redesigned 
student success course, and intrusive advising for students, along with a locally developed 
software application for tracking student progression and improving advising. As a result of 
integrating interventions into a more comprehensive model that supports students along a 
pathway from intake to the accumulation of 30 college credits, the college has seen some initial 
success in improving the experience of first-year students. According to one administrator, “The 
combination of strategies appears to be working, and we are continuing to look at ways to shore 
up the advising process. The sum total of our efforts has led us to some success in getting 
students to their 30 credits [in their first year].” This idea of the effectiveness of implementing 
complementary sets of innovative practices is consistent with research on high-performing 
organizations (Jenkins, 2011). 

In contrast with this more comprehensive approach to redesigning the intake and 
advising process, two colleges — WA-D and WA-E — sought to improve the experience of 
new students by revising student orientation. At the core of these efforts were changes to 
institutional policy, including making orientation mandatory and eliminating late registration for 
first-time students. The colleges reported that the changes to the intake process reached a large 
number of students and increased first-to-second-term retention rates. However, although large 
in scale, these redesigned orientation programs involved relatively brief contact with students, 
raising the question of how much benefit beyond first-to-second-term retention they may have 
had. 


The major student services reform at WA-A was the development of a transition center 
for ESL students. Recognizing that an extremely low number of ESL students were successfully 
transitioning to a college-level program, the college developed a one-stop center offering help 
with advising, financial aid, and academic support to facilitate the transition from ESL courses 
to college-level programs. The center specifically targeted students at the higher levels in the 
ESL course sequence. The transition center reached an increasingly larger number of the target 
population in each year after initial implementation: rates of participation jumped from 47 
percent of the target population in AY 2007-2008 to 74 percent in AY 2008-2009. Moreover, 
transition rates to college-level coursework also increased dramatically: in AY 2007-2008, 12 
percent of the target population transitioned, and in AY 2008-2009 the transition rate increased 
to 20 percent. According to one administrator, the college’s approach to serving ESL students 
“is very different from how it was looked at five years ago,” mostly as a result of the ATD 
work. 


39 



Instructional Programs 

In addition to redesigning student services, some of the Washington colleges also 
sought to remake instructional programs. Their efforts tended to focus on developmental 
education sequences (as mentioned, five of the six colleges reported implementing at least one 
strategy that targeted students in developmental education courses). Several of the changes were 
structural and focused on reducing (or, at one college, increasing) the number of required 
courses in the developmental sequence, developing modularized courses, integrating student 
success content into existing courses, or implementing a cohort model. 

Developmental math sequences in particular were a focus of Achieving the Dream 
work. Of the 15 strategies related to developmental education across the Washington colleges, 
1 1 targeted developmental math. For example, following the Statway model developed by the 
Carnegie Foundation for the Advancement of Teaching, 8 WA-A transformed its developmental 
math sequence to make the math curriculum more relevant to students’ academic pathways. In 
addition to the curricular reform, the strategy reduced the length of the developmental math 
sequence from three terms to two terms for students not pursuing STEM programs of study. 
Administrators reported that because the entire developmental math structure had changed, 
faculty became engaged in deeper conversations about math content and placement procedures 
more generally. These changes were not fully implemented until spring 2010. According to the 
college, one year later, as of spring 2011, 75 percent of the roughly 3,000 developmental math 
students served annually had gone through the new sequence. 

The majority of redesign efforts in both instruction and student services targeted 
students in developmental education or gatekeeper courses, first-year students, or some 
combination of these groups. As a result, colleges reported improvements in early retention 
rates and in completion of developmental and gatekeeper courses. In the concluding chapter we 
argue that for colleges to experience improvements in other measures of performance, 
particularly completion, they will need to undertake similar changes to degree programs. 

Engagement of Faculty to Develop and Disseminate Instructional 
Innovations 

Research on effective organizations inside and outside education indicates the 
importance of engaging front-line employees in the improvement process. In Washington, four 
of the ATD colleges — WA-B, WA-C, WA-D, and WA-F — sought to engage faculty in 
developing and diffusing innovations in instruction, with varying degrees of success. We 


8 For more information, see the Carnegie Foundation for the Advancement of Teaching’s website: 
http://www.camegiefoundation.org/statway. 
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describe these efforts below to identify the conditions under which substantial changes are 
likely to occur. 

College leaders can promote faculty engagement by facilitating the dissemination of 
innovative practices through professional development opportunities and the sharing of 
knowledge among faculty and staff across the institution. For example, WA-C implemented an 
instructional reform using the Universal Design for Learning (UDL) framework. UDL is a 
professional development program that uses research on cognitive science and student learning 
styles to improve curriculum design, pedagogy, and assessment. At WA-C, the intervention 
calls for full-time and adjunct instructors to form cohorts of 12 to 15 faculty members who work 
together over three terms to create and implement UDL strategies. The college found that 
course completion rates improved by 3 percent or more for students of instructors who 
participated in UDL. Initially, the intervention was piloted for faculty teaching developmental 
education and gatekeeper courses, but it has expanded over time to include faculty members 
from other academic departments. Initial training in UDL created faculty “champions” for the 
approach. Each term a faculty member was given full-time release to coordinate UDL training 
activities. 

While not mandatory, UDL has slowly gained traction at WA-C. As of spring 2011, 
over 60 of the more than 400 full- and part-time faculty members had participated in creating 
and implementing UDL strategies in their classes (primarily in developmental-level and 
gatekeeper courses). The college is planning to further integrate the intervention in other 
disciplines as part of college- wide professional development opportunities. 

Another organizational practice that encourages broad-scale instructional reform is 
taking a faculty-driven approach to developing reforms. The most successful direct student 
intervention developed under Achieving the Dream at WA-B was a Reading Apprenticeship 
(RA) program that was modeled on the approach developed by WestEd. RA is designed to 
improve student learning through a structured approach to reading and understanding texts in 
any discipline. At WA-B, a faculty member learned about the Reading Apprenticeship strategy 
at an ATD Strategy Institute and introduced the concept to faculty, administrators, and the ATD 
core team at the college. The strategy was developed by a team of faculty and staff members 
that described their process as a “grassroots movement,” contrasting it with the “top-down” 
approach initially used for ATD strategy development at the college. 

After seeing initial improvements in first-to-second quarter retention for students 
participating in the intervention and gains in reading on the Comprehensive Adult Student 
Assessment Systems (CASAS) 9 assessment for basic skills students, the strategy leadership 
team increased its outreach to involve more faculty members in disciplines across the college 


9 For more information, see CASAS’ website: www.casas.org. 
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and drafted a comprehensive plan for scaling up the program. With the support of the 
administration, the strategy team developed a Reading Apprenticeship website, disseminated a 
monthly program newsletter, recruited new participants at department meetings, and created 
posters and tee shirts promoting the program. The team also embraced the use of data to gather 
evidence of the program’s effectiveness and made this information widely available at the 
college. As a result of these efforts, senior leaders recognized that having empowered, 
motivated faculty and staff as change agents increased buy-in among other faculty. 

As one administrator noted, “When you identify a strategy, you need to have a 
champion for it. Reading Apprenticeship is successful because our [faculty member] strategy 
leader is invested in it.” Similarly, a student services staff member said, “Reading 
Apprenticeship had a slow start, but now people are talking about it all over campus. Anytime 
you get that kind of enthusiasm, you are well on your way.” Encouraged by growing 
enthusiasm and the support of the administration, the strategy leaders reported that increasing 
numbers of faculty were participating in Reading Apprenticeship professional development 
activities. By spring 2011, over 130 faculty members had participated in training in the RA 
method. Based on this enthusiasm, the college’s director of institutional research estimated that 
in spring 2011 38 percent of all students and 46 percent of students in professional technical 
courses were taught by instructors who had at least two hours of RA training. No formal 
assessment has been done on how much these instructors use RA in their classes, however. 

The RA intervention suggests that the engagement of faculty in the knowledge building 
and ownership of reform efforts is an important component of implementing changes in 
classroom practice at scale. However, without the active support of administrators to facilitate 
communication and spread reform efforts, broad faculty engagement and scaling-up efforts are 
likely to be more difficult to achieve. For example, another effort to foster instructional 
innovation among faculty was an integrative learning program implemented at WA-F. 
Integrative learning engages faculty from multiple disciplines in a common theme for their 
coursework to enrich student learning across courses. Similar to the Reading Apprenticeship 
program, the integrative learning strategy was driven by a core group of motivated faculty. A 
faculty member noted, “You get into a sticky situation sometimes when the administration gets 
too involved, because then the expectation is that you have to be involved in this. I think one 
thing that has made [integrative learning] so powerful is that it is totally voluntary for faculty.” 
In contrast with the interventions described above, however, faculty members involved in 
integrative learning described a lack of administrative support for their efforts to innovate and 
scale up their practices. An instructor stated: 
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I think Achieving the Dream was very focused on student success, but it 
wasn’t focused enough on the systemic support systems wi t hin the college 
itself. If we don’t look at what is supporting faculty and what is supporting 
the cross-pollination and sharing across the community, then I think you are 
missing a huge part of what makes a college successful. As an Achieving the 
Dream college, there were some attempts to bring faculty together, but it felt 
superficial. 

Without the infrastructure to support the sharing of innovative instructional practices, 
there were limited opportunities to extend the integrative learning strategy to additional faculty 
across the college, and as a result the strategy was only being used by a small group of 
instructors. As of spring 2011, about 150 students were participating in the intervention each 
term, which is approximately 2 percent of the total enrollment and 22 percent of students 
enrolled in the targeted courses. 

At WA-D, administrators were resolved to let math faculty lead the college’s efforts to 
revamp the developmental math curriculum, and faculty were empowered to develop and test 
different approaches. However, the math faculty was cautious about adopting approaches for 
which there was not sufficient supporting evidence. The college initially lacked a strong 
institutional research function that resulted in widespread mistmst of data and a culture not 
conducive to supporting change and innovation (see Chapter 2 for more information on 
WA-D’s experience). As a result, changes to the developmental math sequence at the college 
have been slow to develop. The interventions described above suggest that the diffusion of 
instructional reforms is enabled by several organizational practices. First, colleges should 
embrace a faculty-driven approach to designing and implementing reforms, especially those 
related to instruction, rather than a top-down approach. Indeed, a grassroots approach is 
consistent with research suggesting that faculty and staff must be centrally engaged in the 
process of improving institutional performance. Second, colleges should support the spread of 
innovative practices by providing the infrastructure and opportunities for faculty and staff across 
the institution to share their knowledge and authority. Third, colleges should ensure that faculty 
and staff understand and believe in the goals of organizational reform. Research suggests that 
these practices are more likely to result in instructional reforms that are sustainable and scalable. 


Innovating at Scale Versus Scaling Innovations 

As a group, the Washington colleges seemed to focus more on improving instruction 
(as opposed to only student services and academic support) than did the Round 1 colleges, with 
at least two implementing strategies almost entirely related to instruction. 
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Research on high performing organizations indicates that effective organizations 
implement integrated, complementary practices that align with organizational goals. Some of 
the Washington colleges did seek to move beyond implementation of discrete interventions to 
make systemic changes in programs and practices, and several of them succeeded. Some of 
these same colleges also succeeded in promoting diffusion of instructional innovations. In those 
cases, the colleges seemed to provide a balance between empowering faculty and leaving them 
alone to try out innovations while at the same time providing support for professional 
development. 

The Washington colleges that were most likely to make systemic changes by 
redesigning programs and services and supporting the diffusion of instructional innovations 
were those that we found to be further along in implementing the Achieving the Dream culture 
of evidence principles. This finding supports the idea, central to the Achieving the Dream 
theory of change, that building a culture of evidence creates the organizational conditions for 
reforming academics and student services to better serve students. 

The approach taken by these colleges was not to scale up innovations per se, although 
they did experiment with different approaches, but to redesign programs and services at scale — 
essentially to innovate at scale rather than to scale innovations. Such innovations are easier to 
sustain since they become part of the way a colleges operates. We also hypothesize that the 
colleges that were able to innovate at scale are those that will see the greatest improvements in 
performance. 

However, even in these more innovative colleges, efforts to redesign programs and 
services in Washington were largely confined to the intake process and developmental 
education. CCRC research suggests that colleges need to take a similar approach to college- 
level programs. In particular, they should take steps to ensure that their programs are well 
structured and well aligned with requirements for the successful transition to further education 
and employment, and that mechanisms are in place to help students enter and complete 
programs of study as soon as possible (Jenkins & Cho, 2012). We hypothesize that the systemic 
changes implemented by some of the colleges might lead to improved retention and progression 
to college-level gatekeeper courses. However, without similar systemic changes to college 
academic programs, we do not see how the efforts implemented by the Washington colleges 
alone can increase student completion and success. 
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Chapter 4 

Use of Data by Faculty and Administrators 


This chapter presents findings from surveys of data use by community college faculty 
and administrators. These respondents also reported on their perceptions of the use of student 
data by their academic and administrative departments (by administrative departments, we 
mean functional areas such as president’s office, admissions, IR, IT, etc.) and by each of their 
colleges as a whole. Our analysis focused on changes in the use of data over three years at each 
of the six Washington colleges that joined ATD in the 2006-2007 academic year. We explored 
changes in the frequency and nature of data use between the first survey, conducted during the 
2007-2008 academic year, and the second survey, conducted during the 2010-2011 academic 
year. 


Our analysis yielded the following main findings, which are generally consistent with 
the findings from the field research: 

• There were positive changes in the extent of use of data over time by faculty 
for teaching-related purposes, particularly at three of the colleges. 

• We observed no changes in the frequency with which faculty used data on 
academic achievement gaps between students of different races/ethnicities 
and income levels. 

• Faculty at three of the colleges perceived an increase in the use of data by 
their academic departments, particularly for curricular decisions and 
identifying courses with high failure rates. 

• The perceived use of data by administrative departments at the colleges 
declined over the time between the two surveys. This finding suggests that 
use of data for planning purposes waned following the first year or two of 
participation in Achieving the Dream, when colleges were strongly 
encouraged and helped to analyze data on student success. 

The findings suggest that there were some positive changes in attitudes toward and use 
of data, but perhaps not as many as we expected at the outset of this initiative, which has among 
its core principles the creation of a culture of evidence to improve student outcomes. Although 
we are encouraged by the generally positive changes that have occurred, the patterns of 
improved data use varied across colleges and within colleges between faculty and 
administrators. 
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Research Methods 


Sample 

The sampling frame included all full-time administrators at the director level or above 
and all full-time faculty. The final sample size for the 2010-201 1 survey was 679: 424 faculty, 
131 administrators, and 124 student services staff. The overall response rate was 69 percent. 
WA-A had the highest combined response rate: 79 percent. The response rate for WA-D and 
WA-E was 73 percent; for WA-B, 69 percent; and for WA-C, 66 percent. WA-F had the lowest 
combined response rate: 58 percent. Response rates for faculty and administrators in each 
survey are provided in Table 4.1. 


Achieving the Dream Colleges in Washington State 
Table 4.1 

Response Rates for Each Survey by College 


College 

2007-2008 Survey 

2010-2011 Survey 

Faculty (%) 

Adm. (%) 

Faculty (%) 

Adm. (%) 

WA-A 

45 

61 

69 

94 

WA-B 

64 

93 

67 

79 

WA-C 

66 

77 

64 

77 

WA-D 

74 

80 

79 

72 

WA-E 

74 

93 

62 

85 

WA-F 

46 

81 

49 

91 

Mean 

62 

81 

65 

83 


Survey Design 

The surveys used a panel design to enable tracking of behaviors and perceptions over 
time. The CCRC-MDRC research team developed two instruments in 2007 : a faculty survey 
and an administrator survey. Designed in consultation with leadership and data coaches working 
with the colleges, we revised the instruments after pilot testing at three ATD colleges that did 
not participate in the study. 

In 2010 we modified the faculty and administrator instruments based on our 
experiences administering the 2007 survey and to account for the greater attention being paid to 
assessment in Washington since then. Our goal was to capture more details. For example, we 
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changed a yes/no question on the frequency or extent of data use to a question whose answer 
used a 7-point Likert scale. We also added a few questions to obtain information that we had not 
elicited in 2007-2008. Further, we added a student services instrument in 2010, based on the 
administrator instrument. Because we did not solicit the views of student services personnel in 
2007-2008, an analysis of the 2010 data is not included in this chapter, which examines 
changes over time. (Analyses of data on the views of student services personnel will be 
presented in a separate, more comprehensive report on the survey results.) 

Each instrument contained approximately 100 items, most of which overlapped, 
relating to three topics: data accessibility, data use, and familiarity with and involvement in 
Achieving the Dream. The instruments also included a set of demographic questions about the 
respondent and his or her role at the college. Most items used a 7-point Likert scale; some 
questions used a yes/no format. 

Both the 2006-2007 and 2010-2011 survey instruments were administered by the 
Human Resources Research Organization (HumRRO), which developed the web interface; 
managed the distribution and administration of the survey, both electronically and in paper 
form; and served as the primary contact for questions and technical difficulties. MDRC and 
CCRC collaborated with HumRRO during the administration of the survey and were 
responsible for the coordination with participating colleges and questions about content. 

Data Analysis 

The survey responses were analyzed to ascertain whether changes occurred between the 
2007-2008 and 2010-2011 surveys in the frequency and extent of data use by faculty and 
administrators at the participating colleges. We conducted a separate analysis for each college, 
and the results were compared across colleges. To capture differences within colleges between 
the years covered by each survey, we used chi-squared and Wilcoxon-Mann- Whitney tests. 
When the median remained the same but the distribution was statistically different, we also 
examined the frequency distributions and percentage distributions of those variables. 

Another set of variables were measured on a Likert scale and captured perspectives on 
data, extent of data use, barriers to data use, and involvement in Achieving the Dream. These 
data were explored using /-tests to compare differences in the means between the two waves of 
the surveys. 
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Extent of Data Use 

In general, the survey results suggest that there were few changes in the use of 
data by faculty and administrators between 2007-2008 and 2010-2011, although some 
positive changes occurred at three or four of the colleges. 

Because most changes were not statistically significant, we only provide items of 
particular interest in tables (in Appendix B), such as faculty use of data to inform teaching- 
related practices (Appendix Table B.3). Summaries of survey responses are provided in 
Appendix Table B.l (for faculty) and Appendix Table B.2 (for administrators). 

Data Accessibility 

Between 2007-2008 and 2010-2011, the reported accessibility of data improved 
among faculty at four of the colleges and among administrators at three colleges. 

At four colleges — WA-A, WA-B, WA-D, and WA-F — faculty responses about the 
accessibility of data improved in at least two of the following areas: accuracy, user friendliness, 
responsiveness of the IR office, sufficiency of IR staffing, and timeliness of data (see Appendix 
Table B.l). WA-A’s faculty responses showed an increase in the user friendliness of the 
formatting of data and an improvement in the timeliness of data. WA-B's faculty responses also 
showed an improvement in the timeliness of data and suggested that the IR office had improved 
its responsiveness. 

For administrators, the reported accessibility increased at three colleges (see Appendix 
Table B.2). WA-B had the greatest number of areas of increase, including accuracy, formatting, 
responsiveness of the IR office, clarity of college reports, and timeliness of data. Data from 
WA-C suggests increases in accessibility in the areas of formatting, IR responsiveness, IR 
staffing, and timeliness of data. Finally, there was an increase in the reported user-friendliness 
of data at WA-D. 

Use of Data by Faculty and Administrators 

Frequency of Data Use by Type 

We asked about the frequency with which faculty and administrators reviewed or used 
data of certain types, including student outcomes and data disaggregated by race/ethnicity or 
income. The scale was as follows: 1 (never), 2 (once every two to three years), 3 (once every 
one to two years), 4 (once per year), 5 (two to three times per year), 6 (four to five times per 
year) and 7 (more than six times per year). Across the 17 types of information we asked about, 
the six colleges demonstrated few statistically significant changes in the frequency of data use 
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between the 2007-2008 and 2010-2011 surveys (not shown in tables). The most changes 
occurred at WA-E, which had statistically significant changes in seven areas, six of which were 
related to administrators’ use of various types of data. Below, we focus on statistically 
significant (p < .05) changes in the use of data by faculty. 

Faculty at WA-D increased their use of data on placement scores and grades to more 
than six times per year from once a year. The faculty at WA-E increased the frequency with 
which they used placement test scores from a median of two to three times per year to four to 
five times per year. We also saw increases in the use of research: WA-B faculty increased the 
frequency with which they used outside research on effective practices, and WA-F faculty 
increased their use of data from qualitative data sources. Although not the focus of Achieving 
the Dream, the most consistent increase in data use across the colleges was in the area of faculty 
use of budget and finance information, with statistically significant increases at four of the six 
colleges: WA-A, WA-C, WA-D, and WA-F. 

Across the colleges, for both faculty and administrators, there were no statistically 
significant changes in the frequency of use of data on course evaluations, other learning 
measures, results from external surveys such as the Community College Survey of Student 
Engagement, and information broken down by income or receipt of financial aid. 

Use of Data by Faculty in Teaching-Related Decisions 

We examined not only the frequency of data use generally but also the extent of data 
use for teaching-related decisions. For these questions, the scale ranged from 1 (not at all) to 7 
(to a great extent). Appendix Table B.3 shows our findings. Two colleges — WA-C and WA-D 
— demonstrated statistically significant increases in the reported extent to which faculty were 
using data on students in their decisions about curriculum, teaching practices, advising students, 
and identifying students who were struggling academically. At WA-A, there were increases in 
the reported extent to which faculty were using data and research on students in decisions about 
curriculum, advising students, and identifying students who were struggling academically, but 
there was no statistically significant change in their use of data for teaching practices. The 
remaining three colleges had few or no statistically significant changes in these areas. 

Sources of Data on Students 

Faculty use of various sources of data generally remained unchanged or declined over 
the three years between the surveys. There were decreases in the use of college reports by 
faculty at two colleges: WA-A and WA-F. There was also a decrease in the reported use of 
student surveys by WA-B faculty. The only increase in the use of a data source was in the use of 
state databases by WA-A faculty. 
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Table 4.2 displays the top three sources of data on groups of students by college. For 
faculty at most of the colleges, requests to the IR/IT office and college-produced reports were 
the primary sources of data. Also noteworthy is that at four of the six colleges, faculty agreed 
with the statement that they “generally do not need information on groups of students.” This is 
in contrast with administrators, for whom that response was not as common (see Table 4.3). 

As with the faculty, the sources that administrators used for gathering data on groups of 
students did not change significantly between the 2006-2007 and 2010-2011 surveys. In 
contrast with the faculty, administrators at three colleges increased their reported use of various 
sources of data on groups of students. In particular, WA-D' s administrators increased their use 
of the college website, college reports, requests to the IR department, and student surveys. 
WA-A’s administrators increased their use of their college’s student information system (SIS), 
and WA-E' s administrators increased their requests to the IR office. The only decrease was in 
administrators’ use of college reports at WA-C. Administrators relied primarily on requests to 
the IR and IT offices and college reports for information. In terms of their top three data 
sources, administrators were less likely than faculty to report that they did not need information 
on groups of students; administrators at WA-F and WA-E also listed the SIS among their top 
sources (see Table 4.3). 


Achieving the Dream Colleges in Washington State 
Table 4.2 

Faculty’s Top Three Sources of Data on Groups of Students by College 


Date 

Source 

Rank 

WA-A 

WA-B 

WA-C 

WA-D 

WA-E 

WA-F 

1 

Requests to 
IR/IT 

Generally do 
not need info 
on groups of 
students 

Student 
surveys or 
focus groups 

My 

department' s 
database 

College 

reports 

Requests to 
IR/IT 

2 

Generally do 
not need info 
on groups of 
students 

Requests to 
IR/IT 

Generally do 
not need info 
on groups of 
students 

Requests to 
IR/IT 

Requests to 
IR/IT 

Generally do 
not need info 
on groups of 
students 

3 

College 

reports 

College 

reports 

Requests to 
IR/IT 

College 

reports 

My 

department' s 
database 

My 

department’s 

database 
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Achieving the Dream Colleges in Washington State 


Table 4.3 


Administrators’ Top Three Sources of Data on Groups of Students by College 


Date 

Source 

Rank 

WA-A 

WA-B 

WA-C 

WA-D 

WA-E 

WA-F 

1 

Requests to 

Requests to 

College 

Requests to 

College 

Requests to 


IR/IT 

IR/IT 

reports 

IR/IT 

reports 

IR/IT 

2 

College 

College 

Requests to 

College 

Requests to 

College 


reports 

reports 

IR/IT 

reports 

IR/IT 

reports 

3 

College web 

site/fact 

book 

State 

databases or 

research 

reports 

My 

department’s 

database 

College web 

site/fact 

book 

SIS 

SIS 


Reasons for Not Using Data 

We investigated respondents’ views about specific barriers to using data, including the 
sense that they were too busy with other responsibilities, that they did not have the skills to use 
the data, that they did not trust the data, and that the data were irrelevant to their job 
responsibilities. For these questions, the scale was 1 (not at all), 4 (some), and 7 (to a great 
extent). 


The reported barriers to using data decreased modestly for faculty at two colleges: WA- 
C and WA-F (see Appendix Table B.l). WA-C faculty reported a decrease from 4.2 to 3.5 in 
the extent to which they felt too busy to review and use data to inform their teaching (p < .10). 
Over the three years between surveys, the faculty at WA-F reported a decrease in the view that 
the data were irrelevant to their responsibilities. One increase in perceived barriers occurred at 
WA-A; there was a statistically significant increase, from 2.3 to 2.7, in the view among faculty 
respondents that they did not use data because it was not relevant to their job responsibilities. 

While there were no changes at three colleges — WA-A, WA-B, or WA-F — in 
administrators’ views of barriers to their use of data, there were decreases at the other three 
colleges (see Appendix Table B.2). Administrators at WA-C showed a decrease in the view that 
they did not have the skills and that they did not trust the data. WA-D administrators showed a 
decrease in their view of being too busy, not having the skills, and the irrelevance of the data. 
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Administrators at WA-E showed a decrease over the three years in their view that the data were 
not relevant for their jobs. 

Participation in Organized Discussions on Improving Student Success 

Achieving the Dream encourages colleges to engage faculty, staff, and other college 
personnel in evidence-based discussions on ways to improve student outcomes. Our data 
suggest that there were minimal changes in the participation of faculty and administrators in 
organized discussions at their colleges on topics related to improving student success. Faculty 
across the colleges reported that they participated in such discussions once per year or two to 
three times per year during both 2006-2007 and 2010-2012 surveys. These results suggest that 
discussions about student success occurred on these campuses at least once per year and in 
some cases once per semester. We saw only three statistically significant changes in the amount 
of participation in organized discussions between the surveys (see Appendix Tables B.l and 
B.2): a decrease among WA-A faculty in discussions of achievement gaps, an increase in the 
same area among WA-D administrators, and an increase in discussion about low-income 
students among WA-E administrators. 

Use of Data by Departments 

There were a number of statistically significant changes in respondents’ perceptions of 
the use of data by their departments, although not necessarily in the desired direction (see 
Appendix Tables B.4 and B.5). For survey questions on this topic, the scale was 1 (not at all), 4 
(some), and 7 (to a great extent). 

The greatest number of perceived changes in the extent of data use by academic 
departments, as reported by faculty, took place at WA-A (see Appendix Table B.4). The survey 
data suggest that there were substantial and statistically significant increases in the extent of this 
college’ s use of data on students for curricular decisions, teaching, program planning, program 
evaluations, long-term planning, budgeting and finance, and identification of high failure rate 
courses among academic departments. Faculty at four of the other colleges also reported an 
increase in their department' s use of data to identify high failure rate courses and revise them. 
Only WA-B had no statistically significant changes in the use of data by academic departments. 

Among administrative departments, survey responses suggest that there was a decrease 
in the use of data between 2007-2008 and 2010-2011 (see Appendix Table B.5). At four 
colleges — WA-A, WA-C, WA-D, and WA-F — the changes in the extent of perceived data 
use were negative, large, and statistically significant as they related to curriculum, program 
planning, and program evaluations. Survey results from WA-C also suggest that administrators’ 
use of student data to identify areas needing improvement also declined over the three years 
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between the survey waves. In general, the use of data for long-term planning and budgeting and 
finance did not change over time. 

Use of Data by the Colleges 

Although we saw some changes in the use of data by academic and administrative 
departments, changes in the perceptions of the use of data by the colleges overall were minimal 
(not shown in tables). The largest statistically significant change occurred at WA-E in the 
administrators’ perceptions of their college’s use of outcomes data. In 2007-2008, 70 percent of 
the survey respondents agreed that their college used data on student outcomes, and in 2010- 
2011 all of the respondents agreed (p < .01). At WA-A, the percentage of administrators who 
reported that their college used data on students of color and low-income students increased 
from 67 percent to 92 percent (p < .10). The percentage of WA-A administrators who reported 
that their college used data to inform strategic planning decreased between the two surveys from 
100 percent to 92 percent (p < .10). These changes were the only statistically significant 
changes in the perceptions of the colleges’ use of data. 

Comparison with Findings from Field Research 

We compared the findings on the extent to which colleges adopted practices reflecting 
the ATD culture of evidence principles from the surveys with those from the field research. We 
expected that colleges that made more progress in implementing practices consistent with these 
principles would show more evidence of increased data use in the three-year period between the 
two administrations of the survey. 

Table 4.4 compares key findings from the field research on colleges practices related to 
data and research with findings from the surveys on changes in data use by faculty and 
administrators. In general, the findings from the field research are consistent with those from the 
survey. The colleges that made more progress between 2007-2008 and 2010-2011 in 
implementing practices to strengthen use of data and research for improving student outcomes 
were more likely to demonstrate increased data use by faculty and administrators according to 
the survey data. 

One exception is WA-C, where the field research found evidence of increased IR 
capacity achieved in part through efforts to train faculty and staff to access and use data 
themselves. In contrast, the survey found that while, overall, administrators at the college had 
increased their use of data on students more in 2010-2011, there was no significant change in 
data use by faculty. The field research also found increased attention, led by the college’s 
president, to gaps in student achievement based on race and income. The survey, however, 
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Achieving the Dream Colleges in Washington State 


Table 4.4 

Comparison of Findings From Field Research and Surveys on Changes in Use of Data on Students at 

Washington ATD Colleges 


College 

Change in COE 
Implementation 

Field Research Findings 

Survey Findings 

WA-A 

High to Very high 

• Increased focus on gaps in achievement by 
race/income and basic skills. 

• Increased engagement of faculty in using data for 
improvement. 

• Creation of data team created to supplement IR 
capacity. 

• Increase in faculty discussions of achievement 
gap data. 

• Increase in data use by academic departments. 

• Faculty perception that IR capacity not adequate. 

WA-B 

Medium to High 

• Increased IR capacity. 

• Increase in administrator perceptions of data 
accessibility. 

• Faculty perception that IR capacity is adequate. 

WA-C 

Low to High 

• Increased IR capacity. 

• Training by IT staff of faculty/staff on data use to 
supplement IR capacity. 

• Increased focus on achievement gaps by 
race/income. 

• Increase in administrator use of data on student 
groups, but not faculty use. 

• Faculty perception that IR capacity nol adequate. 

• No increase in faculty/department discussions of 
achievement gap data. 

WA-D 

Low to High 

• Increased faculty data use for instructional 
improvement. 

• Training by IR staff of faculty/staff on data use to 
supplement IR capacity. 

• Increase in data use by individual faculty and 
academic departments for curriculum and 
instruction. 

• Faculty perception that IR capacity is adequate. 

WA-E 

Very low to Low 

• Increased data use by administrators, but not 
faculty. 

• Increase in administrator data use, including data 
on student outcomes. 

• No significant increase in data use by faculty. 

WA-F 

No change 

• IR capacity low. 

• Faculty perception that IR capacity increased. 


NOTE: Italicized survey findings indicate where survey finding does not support finding from field research. Non-italicized survey findings are 
consistent with field research findings. 


54 



found no increase in the frequency with which faculty used data on achievement gaps. The field 
research offers an explanation for this seeming inconsistency: Faculty at the college initially 
resisted use of data for improvement. While that resistance waned over time, some faculty 
members still indicated that ATD is perceived as a top-down initiative at the college and that 
administrators are more invested in ATD than the faculty. 

The field research found that WA-A attempted to supplement the capacity of its IR staff 
through the creation of a data team that helped to promote data use and assisted the IR staff in 
addressing requests for information from faculty and staff. Despite these efforts, the survey 
responses indicate that in 2010-201 1 faculty still perceived IR at the college to be inadequate to 
meet their needs. This finding suggests that while there may have been improvements, the 
college IR staff and data team need to do more to meet the faculty needs for information and 
research. 

At WA-F, the survey results suggest that faculty felt that IR capacity had increased. The 
field research found that the new IR director, hired after a period in which the institution had no 
IR staff, was competent but overwhelmed with multiple responsibilities, including accreditation 
work and supporting the information needs of senior staff, that left little time to work with 
faculty. Faculty may have perceived an increase in IR capacity, but the view of the IR staff and 
administrators is that the IR staff capacity was still not sufficient. 


Implications of Data Use Findings 

By administering the surveys three years apart, we were able to assess changes in 
faculty and administrator practices and perceptions related to the use of data between 2007- 
2008 and 2010-2011. While most of the changes we observed were not statistically significant, 
our analyses did reveal some statistically significant and meaningful changes. 

The data sources on groups of students that faculty and administrators used for were 
mostly unchanged overall; however, there were statistically significant positive changes in the 
extent to which WA-D administrators reported using the college’s website, college reports, 
requests to the IR department, and college administered surveys to gather data on groups of 
students. Likewise, while there were not many changes in the frequency with which faculty and 
administrators used various types of data overall, there were notable changes among faculty at 
WA-A and WA-D and some changes at WA-C and WA-F. 

We did not see any change in the frequency with which faculty used data on academic 
achievement gaps disaggregated by race/ethnicity or income. Given the focus of Achieving the 
Dream on improving student outcomes and narrowing the gap between students of different 
races/ethnicities and income levels, this is a noteworthy finding. However, it is encouraging that 
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there were increases in data use by faculty for teaching related purposes, particularly among the 
faculty at three colleges: WA-A, WA-C, and WA-D. In addition, there was an increase in the 
frequency with which administrators at one college — WA-E — used student outcomes data 
and disaggregated data. 

There were numerous statistically significant positive changes related to the perceived 
use of data by academic departments. We saw noteworthy increases in faculty perceptions of 
the extent of data use by their departments at WA-A, WA-D, and WA-E, particularly for 
informing curricular decisions and identifying courses with high failure rates. WA-A also 
demonstrated statistically significant increases in the extent to which faculty used data for 
teaching, program planning, program evaluations, long-term planning, and budgeting and 
finance. 


Unlike the academic departments, administrative departments demonstrated a decrease 
in the use of data in the three years between surveys. The changes between 2007-2008 and 
2010-2011 at four colleges — WA-A, WA-C, WA-D, and WA-F — were negative and 
statistically significant in the areas of curriculum, program planning, and program evaluations; 
the use of data for long-term planning and budgeting and finance did not change at any of them. 
The overall decline in the use of data by administrative departments revealed by the surveys 
may reflect that data use for planning purposes waned following the first year or two of the 
colleges' participation in Achieving the Dream. During this period, colleges were strongly 
encouraged and helped to analyze data on student success to aid in planning the strategies that 
were implemented in subsequent years. 

The survey results were generally consistent with the findings from the field research. 
Colleges that were found through the field research to have increased their implementation of 
practices related to the use of data were more likely to have survey results showing increased 
data use. Overall, the survey results suggest that while Achieving the Dream may have had 
some positive influence on use of data by faculty and administrators at the Washington colleges, 
the increases were relatively small and not consistent across all of the dimensions of data use 
that we examined. 
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Chapter 5 

Student Outcome Trends 


This chapter analyzes student outcome trends at the six ATD colleges in Washington 
State. The goal of Achieving the Dream is to help colleges focus attention on student data and 
make changes in policies and practices that will improve student outcomes based on these data. 
Using student-level data from the entering fall cohorts of 2003 through 2009, the trends analysis 
presented here examines average initiative-wide outcomes across the Washington colleges on 
five key Achieving the Dream indicators, measured two years after the students entered college: 

• Attempting and completing developmental education (remedial) courses in 
English and mathematics 

• Attempting and completing the first college -level course (gatekeeper) in 
English and mathematics 

• Attempting and completing credits 

• Persisting fall-to-fall and year-to-year 

• Earning certificates or degrees 

The previous chapters presented analyses of the colleges’ progress toward building a 
culture of evidence, the student success strategies implemented in the six colleges, and the 
colleges’ use of data. All are key components of the Achieving the Dream model, and they 
provide a foundation for the colleges’ efforts to improve student outcomes. Those chapters 
suggest that the colleges made some important progress in their efforts, but that the scale of the 
reforms may not yet be large enough to significantly impact college-wide trends. 

The main findings from the student outcomes analysis discussed in this chapter are the 
following: 

• The average student outcomes across the six colleges appear largely 
unchanged after the colleges began implementing Achieving the Dream. 

• The initiative does not appear to have reduced the achievement gap along 
racial or economic lines. The gap between students who received Pell grants 
(a measure of low-income status) and those who did not appears to decrease 
over students’ academic careers, but this was also true prior to 
implementation of Achieving the Dream. 
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There do not appear to be greater gains associated with Achieving the Dream 
for some colleges compared with others. 


These findings should be interpreted cautiously. The Washington colleges implemented 
a wide variety of student success strategies, and they seem to be reaching more students than in 
earlier rounds of the initiative. In this sense, the colleges appear to have built on the experiences 
of previous rounds of Achieving the Dream, and in recent years they have been moving closer 
to affecting large portions of their student populations. With respect to the 2007 through 2009 
cohorts included in this analysis, however, the Washington colleges were still reaching 
relatively small proportions of students through their new strategies. 10 Moreover, few of the 
colleges reached scale with a comprehensive set of strategies that touched students across the 
full range of their experiences, even in more targeted areas of the student experience such as 
intake and remediation, where most of the ATD strategies were focused. It may be that the 
colleges positively impacted students on a smaller scale than these analyses could detect, or that 
they positively impacted larger numbers of students in subsequent years. 


Research Methods 

This chapter uses student data from the ATD database, maintained by JBL Associates, 
that include student-level demographic characteristics and outcomes for all degree- or 
certificate-seeking students entering the institutions for the first time in the fall semester. The 
data do not identify students’ participation in different ATD strategies, but they do identify 
developmental education students in both English and math. For the Washington colleges 
analyzed in this report, the student outcomes data include the entering fall cohorts of students in 
2003 through 2009: four cohorts of students that first enrolled before the initiative began (2003 
through 2006 cohorts), and three cohorts that first enrolled after implementation of Achieving 
the Dream (the 2007 through 2009 cohorts). Two-year outcomes are analyzed for each cohort. 1 1 

We present several figures that illustrate average institutional trends across the 
Washington colleges, where each college’s average outcome is weighted equally in the 
analyses. The analyses presented here are purely descriptive. Each figure shows average student 
outcomes for each of four pre-initiative cohorts prior to colleges implementing Achieving the 
Dream (2003-2006), and three cohorts after implementation (2007-2009). 12 Although the 
analyses do address whether the initiative substantially impacted average student outcomes 


10 The 2007 through 2009 cohorts were the most recent ATD cohorts for which two-year outcomes 
were available for this study. 

11 For the 2009 cohort at WA-B, only one-year outcomes are included, because the college did not 
submit data for 2011. Alternate analyses that exclude WA-B do not lead to substantially different 
conclusions than those described in this chapter. 

12 The figures show unadjusted averages for each outcome in each year. 
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across the six colleges, they do not indicate whether specific Achieving the Dream strategies 
changed student outcomes, nor whether the initiative as a whole impacted student performance 
in Washington on a smaller scale. 

Such conclusions generally require more rigorous methods, such as randomized 
controlled trials in which students or colleges are randomly assigned to a treatment group that 
participates in new ATD programs or to a control group that does not in order to measure the 
counterfactual — or what would have occurred in the absence of the initiative. The colleges, 
however, were not selected randomly, and the goal of Achieving the Dream has been college- 
wide, institutional change that impacts all students. Partly because of this goal, the initiative’s 
leaders determined that randomly assigning students was inappropriate: If the initiative helped 
transform entire colleges, then both groups of students within a college would be affected, 
invalidating the research design (that is, there would be no counterfactual). For some 
evaluations, quasi-experimental methods can provide good estimates of counterfactual 
outcomes and support stronger claims about causal impacts. For Achieving the Dream, 
however, quasi-experimental methods are less useful and do not provide causal estimates. Such 
methods cannot control for many important factors and changes at the colleges that may also 
affect student outcomes, and they could easily suggest incorrect conclusions. 


College-Wide Trends in Student Outcomes 

This section analyzes trends in several important student outcome indicators and 
provides a detailed overview of outcomes for students entering the colleges between 2003 and 
2009. 13 


Developmental Education Referrals 

Figure 5.1 shows the percentages of each cohort referred to developmental education in 
English and math, averaged over the ATD colleges in Washington. 14 While the trends of 
referrals do move up and down in each subject, the results shown in the figures do not suggest 
large systematic changes associated with the initiative. The top panel of Figure 5.1 shows the 
percentages of students referred to developmental English in each year of the study, averaged 
across all institutions. The vertical line at 2006 marks the last entering cohort before the colleges 
began implementing Achieving the Dream. In 2003, the first year of the pre-initiative period, 
about 25 percent of the cohort was referred to developmental education in English. Over the 
next two cohorts, referrals rose to about 29 percent before returning to 25 percent in 2006. In the 


13 Additional trends analyses are provided in Appendix C. 

14 Not all colleges reported data for all outcomes. The figures in this chapter present averages across 
all colleges that reported data on the given outcomes. 
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post-initiative period, the reduction in referrals to developmental English continued into 2007. 
About 20 percent of the 2007 cohort was referred to developmental education, and referrals 
remained steady at about 21 percent for the next two cohorts. The data, however, do not provide 
enough information to determine whether the lower percentages of students assigned to 
developmental English in the post-initiative period represents a continuation of the downward 
trend that started with the 2006 cohort or a departure from the pre-initiative trends associated 
with Achieving the Dream. 

The bottom panel of Figure 5.1 shows the percentages of students referred to 
developmental math in each year of the study, averaged across all institutions. Compared with 
developmental English, many more students were referred to developmental math during the 
study period. The referral pattern is similar to that of developmental English, with a slight 
increase in referrals following the 2003 cohort, from 59 percent to 62 percent in the subsequent 
two cohorts, followed by a reduction in 2006 to 53 percent. As in developmental English, 
referrals to developmental math level off at about 52 percent, before rising to 56 percent in the 
last cohort. 

It is important to note that the majority of ATD strategies in the Washington colleges 
targeted developmental education students, but by 2009, when the last cohort analyzed in this 
study entered college, none of the colleges had implemented strategies at scale that affected all 
developmental education students. Figure 5.1 shows that a substantial portion of all students in 
the Washington colleges were referred to some type of developmental education. Most of these 
students required developmental education in math. Compared with the pre-initiative years, 
somewhat smaller percentages of students in both subjects were referred under Achieving the 
Dream. It could be that reforms related to ATD led to small decreases in the percentages of 
students referred to developmental education, but the changes could also reflect changing 
student populations. For both English and math referrals, the downward referral trend began 
prior to implementation of ATD reforms. As discussed in Chapter 1, each of the colleges 
experienced increasing enrollments over the course of the study, so it is plausible that the 
student populations may have also changed during this period. In general, however, the trends 
depicted in Figure 5.1 suggest that there was not a dramatic change in the proportion of students 
referred to developmental education associated with the implementation of Achieving the 
Dream. 


Developmental Education Completion 

Figure 5.2 shows the completion rates for students referred to developmental education 
sequences in English and math The top panel shows the two-year completion rate for 
developmental English. The rate fluctuated more widely in the pre-initiative period, reaching a 
high of nearly 50 percent completion for the 2004 cohort of students and a low of 39 percent for 
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the 2005 cohort. After the implementation of Achieving the Dream, the completion rate became 
more stable, generally about 42 percent, but the overall average, and the general trend, do not 
differ markedly from the pre-initiative period. On average, the pre-and-post-initiative outcomes 
for developmental education students in English look similar. 

The bottom panel of Figure 5.2 shows the two-year completion rates for developmental 
math. The pre-initiative trend in math completion also oscillates, averaging between 23 percent 
and 28 percent completion. As with developmental English, the trend for developmental math 
remains very similar after implementation of Achieving the Dream. It is relatively stable and 
similar to what was observed before implementation. 

It is important to put these figures in the context of the previous chapters’ findings. 
Most colleges’ strategies had not reached a large proportion of the students before the final 
cohort of students examined here (the 2009 cohort) entered college, although some strategies 
did reach large numbers of students in subsequent years. Chapter 3, for example, described how 
one college transformed its developmental math sequence. But in 2009 that math reform had 
not yet been fully implemented. Two years later, however, in spring 2011, nearly 75 percent of 
developmental math students had gone through the new math sequence, thereby suggesting that 
large-scale reforms that did occur may have primarily impacted cohorts of students that are not 
included in this analysis. 

Strategies that are working on a smaller scale, moreover, may not be evident in the 
average trends depicted here. For example, an impact of 10 percentage points on a 
developmental education target population would be an impressive improvement in the context 
of community college reforms. For the Washington colleges, however, it would most likely be 
invisible in trend lines, such as those in Figure 5.2, because of scaling challenges and other 
factors at the colleges. For example, most strategies reached fewer than 25 percent of their 
target population. An improvement of 10 percentage points for 25 percent of targeted students 
would translate into an increase of only 2.5 percentage points for the entire targeted population 
at that college. If other colleges reached fewer students, the total average increase would be 
further diluted. Random fluctuations in the trends could then swamp meaningful changes in 
student outcomes for smaller subgroups. 
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Achieving the Dream Colleges in Washington State 


Figure 5.1 

Trends in Developmental English and Developmental Math Referrals 
Among All Students: 2003-2009: Washington ATD Colleges 
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SOURCE: MDRC calculations using the Achieving the Dream database maintained by JBl, Associates. 

NOTES: Calculations for these figures used institutional means for all available data for sample members in the fall 2003 through fall 2009 
cohorts at Achieving the Dream Round 3 Washington colleges. The vertical line shows the last cohort prior to the implementation of 
Achieving the Dream. WA-B and WA-D were excluded from the English developmental trend due to lack of data; WA-B was excluded from 
the math developmental trend due to a lack of data. 
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Achieving the Dream Colleges in Washington State 

Figure 5.2 

Trends in Developmental English and Developmental Math Completions 
Among Students Referred to Any Developmental English or Math Course, 2003-2009: 
Two Year Outcomes for the Washington ATD Colleges 
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SOURCE: MDRC calculations using the Achieving the Dream database maintained by JBL Associates. 

NOTES: Calculations for these figures used institutional means for all available data for sample members in the fall 2003 through fall 2009 
cohorts at Achieving the Dream Round 3 Washington colleges. The vertical line shows the last cohort prior to the implementation of 
Achieving the Dream. WA-B and WA-D were excluded from the English developmental trend due to lack of data; WA-B was excluded from 
the math developmental trend due to a lack of data. 
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Gatekeeper Courses 

Considerations about the scale of student improvement strategies become even more 
important when interpreting the next set of figures, where average outcomes extend beyond the 
developmental education populations to include entire cohorts of students in the Washington 
ATD colleges. Here, any impacts for a subset of targeted students would be further diluted than 
those for developmental education students by including non-targeted students in the institution- 
wide averages. Figure 5.3 shows the proportions of all students who passed gatekeeper courses 
(the first college -level courses) in English and math within two years. The top panel shows that 
about 38 percent of students passed gatekeeper English in the 2003 cohort. Over the pre- 
initiative period, this proportion decreased slowly, to 35 percent for the 2006 cohort. By the end 
of the follow-up period, less than 30 percent of the 2009 cohort passed gatekeeper English in 
their first two years in college. 

The bottom panel shows that about 23 percent of students passed gatekeeper math in 
the 2003 cohort. In gatekeeper math, the pass rate also decreased over time, although more 
slowly than for English. By the end of the follow-up period, about 17 percent of the 2009 cohort 
passed gatekeeper math in their first two years in college. 
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Achieving the Dream Colleges in Washington State 

Figure 5.3 

Trends in Gatekeeper English and Gatekeeper Math Completions Among All 
Students, 2003-2009: Two-Year Outcomes for the Washington ATD Colleges 
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SOURCE: MDRC calculations using the Achieving the Dream database maintained by JB1. Associates. 

NOTES: Calculations for these figures used institutional means for all available data for sample members in the fall 2003 through fall 2009 
cohorts at Achieving the Dream Round 3 Washington colleges. The vertical line shows the last cohort prior to the implementation of 
Achieving the Dream. Gatekeeper course completion estimates for the 2009 cohort at WA-B consist of only one year of data. 
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Persistence and College Completion 

Figure 5.4 shows the final two outcomes considered in this chapter: persistence and 
completion. The top panel shows the percentage of each cohort persisting into the fall semester 
following entry. In the pre-initiative period, the percentage of students persisting from fall to fall 
oscillated between 43 percent and 46 percent. After ATD implementation, the trend slowly 
increased. By the 2009 cohort, about 48 percent of the cohort persisted into the subsequent fall. 

As noted in the conclusion to Chapter 3, the analysis of strategies suggests that impacts 
on student outcomes were most likely to be observed in possible greater advancement to 
gatekeeper courses and in improved retention. The trend lines in Figures 5.2 and 5.3 do not 
suggest that students, on average, were making greater progress through the developmental 
sequences and gatekeeper courses, but Chapter 3 also suggested that this finding may be related 
to issues of scale. The top panel of Figure 5.4, however, is more positive. The improvements are 
marginal but would be encouraging if they are due to reforms at the colleges. It is important, 
however, to be cautious with these interpretations. Just as the decreasing trends for the 
gatekeeper courses do not suggest a negative impact due to the initiative, the increasing trend 
for fall-to-fall retention may have other causes. 

Finally, the bottom panel of Figure 5.4 shows the percentages of students who 
completed any credential within two years, averaged across the six colleges. Here, the trend line 
oscillates between 18 percent and 24 percent. As with the other outcomes, the post-initiative 
trend does not differ significantly from the pre -initiative trend. 
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Achieving the Dream Colleges in Washington State 

Figure 5.4 

Trends in Fall-to-Fall Persistence and Completion of Any Credential Among All 
Students, 2003-2009: Two-Year Outcomes for the Washington ATD Colleges 
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SOURCE: MDRC calculations using the Achieving the Dream database maintained by JBL Associates. 

NOTES: Calculations for these figures used institutional means for all available data for sample members in the fall 2003 through fall 2009 
cohorts at Achieving the Dream Round 3 Washington colleges. The vertical line shows the last cohort prior to the implementation of 
Achieving the Dream. WA-B was excluded from the fall-to-fall persistence trend due to lack of data. The estimate for completion of any 
credential for the 2009 cohort at WA-B consists of only one year of data. The 2009 cohort at WA-C is not included in the fall-to-fall estimate 
due to missing data. 
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Subgroup Trends in Student Outcomes Among the Colleges 

An important goal for ATD colleges is the reduction of achievement gaps based on 
students’ race and economic status, and it is useful to know if some colleges have made greater 
strides in comparison to others. Our analyses suggest that on some measures the achievement 
gap lessens over students’ academic careers at the Washington colleges, at least when 
disaggregated by economic indicators. This was true during both the pre-initiative and initiative 
periods, and we did not find evidence that implementation of Achieving the Dream caused these 
changes or otherwise reduced achievement gaps based on race or economic status. 

Figure 5.5 shows the percentages of students referred to developmental English and 
developmental math by Pell grant status, an indicator of low-income status. The top panel 
shows that low-income students generally entered college in greater need of developmental 
education in English than did higher income students. The solid line shows the percentages of 
Pell grant students and the dashed line shows the percentages of non-Pell grant students. The 
percentage of Pell grant students referred to developmental education in English moves up and 
down over the study period, but is consistently higher than the percentage of non-Pell grant 
students. Likewise, the bottom panel shows that Pell grant students are consistently referred to 
developmental math at higher rates in comparison to non-Pell grant students. 

Figure 5.6 shows the percentages of students who completed gatekeeper courses in 
English and math, disaggregated by Pell grant status. In contrast with the higher rates of 
developmental education referral for Pell grant students, Figure 5.6 shows that Pell grant 
students tended to complete gatekeeper courses at least as frequently as non-Pell grant students. 
The top panel shows that Pell grant students generally passed gatekeeper English at similar or 
higher rates when compared with non-Pell grant students. The bottom panel shows that Pell 
grant students consistently passed gatekeeper math at about the same rate as non-Pell grant 
students. 

Figure 5.7 shows the percentage of students who persisted from fall-to-fall and who 
completed any credential within two years, both disaggregated by Pell grant status. The top 
panel shows that, in fact, Pell grant students appear to have had higher retention rates than non- 
Pell grant students, a relationship that also appeared in analyses of Round 1 Achieving the 
Dream colleges (Rutschow et al., 2011). The bottom panel suggests that graduation in two years 
for Pell grant students is usually very close to the completion rate for non-Pell grant students. 
Given that Pell grant students, on average, enter college less prepared than non-Pell grant 
students, these could be encouraging findings. It may also be true, however, that students with 
stronger intentions to stay in school are more likely to apply for and receive Pell grants. At the 
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Washington colleges, Pell grant students are also more likely to attend full time, which may 
indicate a greater likelihood to persist. 

This evaluation also examined outcome trends for the colleges individually; we did not 
find evidence of greater gains associated with Achieving the Dream for some colleges 
compared with others. Appendix D provides figures for individual colleges that compare 
average post-initiative outcomes to average pre-initiative outcomes for each. 
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Achieving the Dream Colleges in Washington State 


Figure 5.5 


Trends in Developmental English and Developmental Math Referrals 
Among All Students by Pell Grant Status, 2003-2009: 
Two-Year Outcomes for the Washington ATD Colleges 
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SOURCE: MDRC calculations using the Achieving the Dream database maintained by JBL Associates. 

NOTES: Calculations for these figures used institutional means for all available data for sample members in the fall 2003 through fall 2009 
cohorts at Achieving the Dream Round 3 Washington colleges. The vertical line shows the last cohort prior to the implementation of 
Achieving the Dream. WA-B and WA-D w ere excluded from the English developmental trend due to lack of data; WA-B was excluded from 
the math developmental trend due to a lack of data. 
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Achieving the Dream Colleges in Washington State 

Figure 5.6 

Trends in Gatekeeper English and Gatekeeper Math Completions 
Among All Students by Pell Grant Status, 2003-2009: 
Two-Year Outcomes for the Washington ATD Colleges 
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SOURCE: MDRC calculations using the Achieving the Dream database maintained by JBL Associates. 

NOTES: Calculations for these figures used institutional means for all available data for sample members in the fall 2003 through fall 2009 
cohorts at Achieving the Dream Round 3 Washington colleges. The vertical line shows the last cohort prior to the implementation of 
Achieving the Dream. Gatekeeper course completion estimates for the 2009 cohort at WA-B consist of only one year of data. 
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Achieving the Dream Colleges in Washington State 
Figure 5.7 

Trends in Fall-to-Fall Persistence and Completion of Any Credential 
Among All Students by Pell Grant Status, 2003-2009: 
Two-Year Outcomes for the Washington ATD Colleges 
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SOURCE: MDRC calculations using the Achieving the Dream database maintained by JBL Associates. 

NOTES: Calculations for these figures used institutional means for all available data for sample members in the fall 2003 through tall 2009 
cohorts at Achieving the Dream Round 3 Washington colleges. The vertical line shows the last cohort prior to the implementation of 
Achieving the Dream. WA-B was excluded from the fall-to-fall persistence trend due to lack of data. The estimate for completion of any 
credential for the 2009 cohort at WA-B consists of only one year of data. The 2009 cohort at WA-C is not included in the fall-to-fall estimate 
due to missing data. 
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Implications of the Outcomes Trends Analyses 

The student outcomes analyses described here evaluate whether Achieving the Dream 
produced large changes in student outcomes at the institutional-level — the primary goal of the 
initiative. The evaluation was designed to answer this question but not to detect smaller 
changes, or even large changes affecting small groups of students. Given how implementation 
proceeded, however, rigorous methods that can detect smaller scale changes may be more 
appropriate for future evaluations. 

The analyses in this chapter do provide a unique look at outcomes trends for seven 
cohorts of students in the six ATD colleges in Washington. The average trends are consistent 
with the findings described in previous chapters. In general, at least during the period for which 
we have data, the colleges’ strategies did not reach a sufficiently large number of students to 
allow detection of changes in institution-wide outcomes, averaged over the six colleges. This 
does not mean, however, that the strategies have not been effective at a smaller scale. There are 
trends that suggest broader change in a positive direction, but they must be considered in light 
of some potentially negative trends in other areas. More broadly, the trends described here show 
where the colleges are doing well, and where they are most in need of improvement. 
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Chapter 6 


Enhancing the Impact of Achieving the Dream in 
Washington State and Beyond 

This report has examined the Achieving the Dream work of six Washington State two- 
year public colleges in the first five years after they joined the initiative in 2006. It is part of an 
overall evaluation of Achieving the Dream funded principally by Lumina Foundation for 
Education, with additional support from College Spark Washington and other funders. It 
provides a follow-up to a spring 2008 study of the early progress of the Washington State and 
Pennsylvania ATD colleges conducted by CCRC and MDRC (Jenkins et al., 2009), assessing 
the progress that the Washington ATD colleges have made since then. It also compares the 
progress of the Washington colleges in building a culture of evidence with that of the first round 
of 26 colleges to join the ATD initiative in 2004. 

In this concluding chapter, we summarize our findings about the progress that the 
Washington colleges have made in building a culture of evidence and implementing strategies 
to improve student success. We specifically examine Achieving the Dream's influence on the 
colleges’ organizational practice; the perceptions of college leaders, faculty, and administrators 
about the benefits of participating in the initiative; their views on the usefulness of the assistance 
provided by the initiative; and other factors that motivated the colleges to increase the use of 
evidence for improving student outcomes. 


Progress of the Washington Colleges in Implementing Systemic 
Reforms 

Comparing the Washington ATD colleges with the Round 1 colleges was of particular 
interest to Lumina Foundation and other funders that wanted to see if the Washington colleges, 
having joined the initiative two years after it began, would progress faster than the colleges that 
joined in the first round. 

As reported in Chapter 2, while it is difficult to compare the six Washington colleges 
with the 26 Round 1 institutions, it does seem that the Washington colleges, as a group, went at 
least as far in building a culture of evidence as the Round 1 colleges. All but one of the 
Washington colleges made progress in implementing the ATD culture of evidence principles. 
Two of them, which early in the initiative had only begun to implement practices consistent 
with the ATD principles, made very substantial progress later. In fact, we rated four of the six 
Washington colleges as having implemented the ATD principles at a high level five years after 
they joined ATD. In comparison, only 1 1 of the 26 Round 1 colleges (42 percent) implemented 
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most of the practices associated with a strong culture of evidence by the end of their five-year 
ATD implementation period. As discussed in Chapter 4, based on the survey we conducted at 
the Washington colleges early in their ATD participation and then again near the end of the 
initial five years, we found only modest increases in the use of data by faculty and 
administrators. However, the largest increases were among colleges that made the most 
progress in implementing culture of evidence principles. 

In Chapter 3 we cited the finding from the MDRC-CCRC report on the Round 1 
colleges that most of the strategies they had implemented for improving student success were 
small in scale (Rutschow et al., 2011). By the end of the five-year period, less than a third (31 
percent) of the strategies were larger scale (reaching at least 25 percent of the target population), 
and those strategies tended to be less intensive. The Washington colleges implemented fewer 
strategies on average, and while the majority were smaller in scale, more than 40 percent were 
implemented on a larger scale. 

The MDRC-CCRC report on the Round 1 colleges highlighted the issue of the scale of 
strategies for improving outcomes, arguing that they were unlikely to be effective unless they 
were implemented at scale. As a result, in this study of the Washington ATD colleges, we took 
a closer look at how colleges that implemented strategies at scale were doing so. We saw efforts 
among some of them to redesign programs and services at scale rather than make attempts to 
bring small interventions to scale. For example, one college redesigned its entire intake and 
advising system to help first-time students choose a program of study and complete a substantial 
number of requirements in their chosen program within the first year. Another created an ESL 
transition center to help adult basic skills students secure financial aid and other supports they 
needed to advance to college-level coursework. The math department in that same college 
restructured the developmental sequence in math following the Carnegie Foundation for the 
Advancement of Teaching’s Statway model to make instruction more relevant to students’ 
specific academic pathway. 

Only about a quarter of the strategies implemented by the Round 1 colleges involved 
changes to instruction as opposed to student services or academic support; in contrast, about 40 
percent of the reforms implemented by the Washington colleges involved changes to curricula 
or teaching methods. At two of the Washington colleges, faculty led successful efforts to spread 
instructional innovations across departments in their institutions. 

As a group, the Washington colleges made progress in building a culture of evidence, 
and some of them were beginning to implement more large-scale or systemic changes in 
programs, instmction, and support services. Many strategies, however, were still reaching a 
relatively small proportion of students. Consequently, it simply may be too early to expect 
large-scale improvements in student success. As with the Round 1 colleges, our analysis of 
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trends in the ATD measures of student progress and completion did not reveal clear patterns of 
improvement for the Washington colleges since they first joined ATD. Some strategies, 
however, may have improved outcomes on a smaller scale than evaluated here. 

While we did see promising changes in practice among the Washington colleges, the 
major changes in instruction and student services were confined to a subset of the colleges, and 
even in them the changes were in their early stages. The major changes in student services 
practices were implemented by WA-C, which revamped its intake and advising process to help 
students get into a program of study as soon as possible. WA-D and WA-E both revamped their 
new student orientation processes, but left the rest of the advising and other services for new 
students mostly unchanged. WA-A established a new center to help adult basic skills students 
transition to college but did not change intake for college-level students. It is important to note 
that the performance measures we examined in the trend analysis in Chapter 5 did not include 
measures of the rate at which adult basic skills students transitioned to college. 

On the instructional front, as described in Chapter 4, WA-A revamped its 
developmental math curriculum to better align with students’ programs of study. These changes 
were not fully implemented until spring 2010 and thus are fairly new; yet according to the 
college, one year later 75 percent of the roughly 3,000 developmental math students served 
annually had gone through the new sequence. Training of faculty at WA-B in the Reading 
Apprenticeship program began in late 2009, and by spring 2011 training had been provided to 
over 130 faculty members who teach nearly 40 percent of all students at the college. On the one 
hand, this percentage reflects an impressive level of participation; on the other, it shows that 
training substantial numbers of faculty in new approaches takes time. Even after two years, the 
majority of faculty had not been trained and no formal assessment had been done on how much 
those who received training actually used the program in their classes. The efforts to train 
faculty in the University Design for Learning (UDL) at WA-C focused on instructors in 
developmental and gatekeeper courses, but as of spring 2011 the majority of the college’s 
faculty had not received UDL training. The changes to instruction implemented by the other 
three colleges were even smaller in scale. 

So, while the Washington colleges as a group are on the right track in focusing more of 
their efforts on systemic reforms that could affect a substantial number of students, at most of 
them these changes were still in relatively early stages. Moreover, among the colleges that have 
implemented reforms at scale, most have done so rather recently — and mostly after the period 
for which we have data on trends in student outcomes (which went through the 2009 academic 
year.) Changing practices at scale across the various phases of the student experience takes 
time, particularly in organizations as complex as community colleges and in the absence of 
strong incentives to change. 
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In the five years after they joined the ATD initiative, four of the six Washington 
colleges had laid the groundwork for systemic reforms in instruction and student services and 
begun implementing them. Respondents at all of the colleges indicated that their institutions 
benefited from participating in ATD, whether by helping them build their institutional research 
capacity, develop a more systematic approach to developing student success strategies, or 
aligning student completion strategies on campus. 


Benefits of Participation in Achieving the Dream 

We asked those we interviewed at the colleges what they saw as the main benefits of 
participating in ATD. Respondents from five of the six colleges indicated that they were doing 
things differently as a result of participating in ATD; only one college (WA-F) indicated that 
ATD has not had an impact on practice at the college. Table 6.1 presents the main benefits and 
indicates the number of colleges mentioning each. 


Achieving the Dream Colleges in Washington State 
Table 6.1 


Perceived Benefits of ATD for Organizational Practice at the Washington Colleges 


Type of Benefit 

Specific Benefit 

Number of 
Colleges 

Institutional Research 
and Data Use 

• Greater access to data and stronger capacity to use 
data in decision-making processes 

5 


• Increased use of data to improve college programs and 
policies 

4 


• Use of data for greater transparency in resource 
allocations/budget decisions 

2 


• Strengthened grant writing capacity 

2 

Student Success 
Strategies 

• Systematic approach to developing strategies based on 
student outcomes data 

4 


• Impetus to develop/expand strategies 

2 

Initiative Alignment 

• Synergy with new accreditation process and with WA 
Student Achievement Initiative 

6 


Respondents at five of the six colleges indicated that their organization benefited from 
increased access to and use of data and research on student outcomes. Three colleges — WA-B, 
WA-D, and WA-E — reported that one benefit of increased IR capacity was a strengthened 
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capacity to pursue grant funding. As the vice president for instruction at one college said, “State 
funding is being cut. The college is more dependent on grant funding, and grants increasingly 
need to be supported with evidence.” This administrator and the president of WA-D also 
emphasized that by helping the colleges increase their IR capacity, ATD has enabled them to 
make budget allocation decisions in a way that is perceived as more transparent and fair. This 
has been especially useful in a time of dramatic cuts in state funding. 

Three of the colleges — WA-A, WA-C, and WA-D — credited their participation in 
ATD with strengthening the process by which they identify ways to improve programs and 
services, and specifically by improving the quality of data for making decisions about what to 
improve and how. All six colleges said that ATD helped them better align efforts to improve 
student completion, particularly those related to meeting the new accreditation standards. 


The Value of Achieving the Dream Assistance 

Leadership Coach and Data Coach 

Table 6.2 summarizes our assessment, based on interviews at the colleges, of the 
perceived usefulness of the ATD leadership coaches and data coaches (or data facilitators). 


Achieving the Dream Colleges in Washington State 
Table 6.2 


Perceived Usefulness of ATD Leadership and Data Coaches 
at the Washington Colleges 


College 

Leadership Coach 

Data Coach 

WA-A 

Low 

High 

WA-B 

High 

High 

WA-C 

High 

High 

WA-D 

High 

Medium 

WA-E 

Low 

Low 

WA-F 

Medium 

Medium 


Two of the colleges found their leadership coach and data coach to be very useful. WA- 
B viewed its leadership coach and data coach as a team and had good rapport with them both. 
The college’s IR director in particular found them especially useful in advising him on the 
various analyses he was conducting to support the ATD efforts and broader student success 
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work at the college. College administrators said they would only continue participating in ATD 
beyond the initial five years if they could have the same coaches. There was broad agreement 
among those we interviewed at WA-C that the both the leadership coach and the data coach 
challenged the college to improve and, as one respondent said, “kept us accountable.” In the last 
couple years, both have sought to focus college leaders on figuring how to sustain both 
successful strategies and the overall work on improving outcomes after the initial five years in 
ATD. 


Respondents at WA-D were generally positive about their coaches and agreed that they 
asked good questions and held the college accountable. Several respondents said that the 
coaches helped overcome the president’ s skepticism about data and convinced the president to 
champion evidence-based decision making at the college. Staff in the college’s Office of 
Institutional Effectiveness said that while the data coach was helpful on overall strategy for 
increasing access to useful information, they would have liked to have more help on the 
technical issues they confronted as they sought to build an IR function at the college from 
scratch. 


Those we interviewed at WA-A were united in saying that the college’ s data coach was 
very useful. Respondents said that the data coach helped frame the data analyses by asking the 
right questions and facilitate the process of inquiry and discussions of data by the college’s 
Achieving the Dream project team. At least one respondent referred to the data coach as 
“invaluable.” College leaders were enthusiastic about continuing to work with that coach. The 
college found the leadership coach less helpful, saying that the coach did not understand the 
college’s culture and was more accustomed to a top-down approach to decision making and 
management. 

ATD project staff at WA-F were generally positive about both the coaches. However, 
the IR director expressed some frustration that they asked for data the college could not easily 
access. Some faculty we interviewed complained that the leadership coach and data coach 
pressured the college to show results, and as a consequence some of the interventions tried were 
evaluated too quickly, before the faculty implementing them had time to work out the bugs. 

The leadership coach and data coach for WA-E did not seem to be a good fit for the 
college. Project staff said that they were more focused on strengthening the college’s data 
systems than on the big picture. The IR staff said that the data coach was not responsive to 
queries when he/she was not on campus for a site visit. Student services staff members hoped 
they would do more to push the college leadership to focus on gaps in achievement of Hispanic 
students and to make more substantial changes. 
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Strategy Institutes 

Most of those who had attended at least one ATD Strategy Institute had positive 
opinions, saying that they were good venues for networking with other colleges engaged in 
similar work and a useful source of new ideas. At least one of the college reported using 
strategies introduced at a Strategy Institute. 

An academic vice president from one college said that the Strategy Institutes were more 
useful when the colleges were first learning about the ATD approach, though later Institutes 
provided a good introduction to the ATD philosophy for new hires. The IR director from 
another college agreed that they were more helpful initially as the college was learning the ATD 
approach and suggested that there should be more opportunities for experienced colleges to 
interact and learn from one another “on advanced topics.” A faculty member at another college 
said that there was too much focus on data and not enough on instruction at the Institutes. The 
president of yet another college argued that ATD needs to find a “structured and accessible” 
way for colleges to learn from each another in addition to the annual Strategy Institutes. 


Other Influences on Organizational Change 

In our interviews at the colleges we sought to get a sense of how much the changes in 
organizational practice we observed were a result of the college’s participation in ATD as 
opposed to some other factors. 

Respondents at every college said that, in addition to Achieving the Dream, the impetus 
for making these changes came from new accreditation standards established by the Northwest 
Commission on Colleges and Universities, which required colleges to set measureable goals for 
improving student outcomes, use evidence for decision making, and continuously improve 
programs and services. Our interviews at WA-C and WA-D indicated that participating in ATD 
allowed the colleges to respond to the new accreditation requirements much more easily. 

Respondents at at least two of the colleges WA-A and WA-B — also mentioned 

the influence of the Student Achievement Initiative (SAI), a state performance funding system 
through which colleges receive incentive funding for increasing the rate at which students reach 
key achievement points . 15 This was particularly the case at WA-A, where faculty not involved 
with ATD conflated ATD with SAI. 

Our assessment is that ATD had the greatest impact on organizational practice at WA-C 
and WA-D. WA-A and WA-B also made substantial changes in practice during this period that 


15 For more information, see the State Board for Community and Technical College’s website: 
http://sbctc.edu/college/e_studentachievement.aspx. See also the reports on Washington State’s Student 
Achievement Initiative by Jenkins, Ellwein, and Boswell (2009) and Shulock and Jenkins (201 1). 
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were consistent with the ATD model. Interviews at WA-A suggest that ATD provided a 
framework and approach to making the changes in practice that the college believed were 
necessary to meet the new requirements of accreditation and the Student Achievement 
Initiative. Administrators at WA-B also said that ATD helped the college build its data capacity, 
but accreditation and the cuts in state funding provided the main impetus for these changes. 


Lessons for Other Colleges 

Encouraged by the accomplishments of the initial six ATD colleges in the state, in 201 1 
College Spark Washington provided funding for an additional ten two-year colleges to join 
Achieving the Dream. College Spark recognizes that the new colleges have a lot to learn from 
the initial six and has asked the experienced colleges to coach the “rookies.” 

Based on our in-depth analysis of the experience of the six original Washington ATD 
colleges during their first five years in the initiative, we highlight several lessons for the new 
Washington ATD colleges and others elsewhere seeking to implement similar reforms. 

Redesign Programs and Supports at Scale 

Research on high performance organizations both in and outside of education indicates 
that improving organizational performance generally requires systemic changes in practice. 
High-performing organizations continually review and, as necessary, redesign their practices to 
ensure that they are well aligned to achieve organizational goals. Some of the six Washington 
ATD colleges ended the planning year with long lists of relatively small interventions they 
intended to implement as part of the initiative. As they began implementation, however, they 
decided instead to focus on a smaller number of efforts that could be implemented at scale. So, 
for example, WA-C redesigned its entire intake and advising system to help students choose 
and enter a program of study quickly, and WA-A restructured its developmental and college- 
level math sequences to better align with students’ intended programs. 

Of course, small-scale piloting and evaluation are often appropriate. However, colleges 
should not waste valuable resources testing approaches that are unlikely to be scalable or 
sustainable over time with their existing operating resources. Moreover, it is not enough to scale 
discrete innovations. Instead, colleges need to rethink how innovations fit within their overall 
policies and practices in a given area and how they might need to be redesigned or restructured 
to better achieve the desired outcomes with existing resources. 
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Involve Faculty From the Start in Leading Efforts to Rethink Curriculum 
and Instruction 

Community colleges are unlikely to see substantial improvements in student completion 
and learning unless there are major changes in classroom practice. Classroom practices are not 
going to change unless the faculty are convinced of the need to make changes and are supported 
to explore possibilities and then implement the most promising. It is critical, therefore, for 
colleges seeking to improve programs and instruction to involve faculty from the start in leading 
reforms. WA-A, the Washington college that succeeded in making the largest change to 
instructional practice — completely revamping its developmental mathematics curriculum — 
benefited from a history of faculty-level innovation and “bottom-up” decision making at the 
institution. The widespread adoption of the Reading Apprenticeship program for improving 
students' understanding of texts at WA-B was spearheaded by a group of faculty as a grassroots 
effort that, if not initiated exactly as a protest against the perceived top-down approach first 
taken by the college in managing ATD, nevertheless was seen by faculty as a contrast to it. 
Similarly, faculty at other colleges also led efforts to improve instruction, although perhaps with 
not as much success as the two examples described here. 

Colleges that want to engage faculty in reforms should do so at the outset, being clear 
about the desired goals, but empowering them to decide, in conjunction with their student 
support services colleagues, how best to achieve the goals. Colleges should also support the 
spread of innovative practices by providing the infrastructure and opportunities for faculty and 
staff across the institution to share their knowledge and experience. 

Strengthen Pathways Into and Through Programs of Study 

As discussed in Chapter 4, efforts by the Washington ATD colleges to redesign 
programs and services were mostly confined to the intake process and developmental 
education. Over time, as the colleges succeed in implementing these reforms at scale, 
we hypothesize that the changes will lead to improved success in developmental courses 
and progress toward college-level gatekeeper courses. As we pointed out in Chapter 5 
on the trend analysis, most of these reforms were implemented at scale after 2009, so 
their effects are not likely to be reflected in the data we have on student progress, which 
only extend to the 2009 first-time fall cohort. In the longer term, however, without 
systemic changes to colleges’ academic programs, we do not see how “front-end” 
reforms of the sort mainly implemented by the Washington colleges alone can 
substantially increase student completion and success. 

Research suggests that a similar systemic reform may also need to be applied to 
college-level programs to ensure that they are well structured and aligned with 
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requirements for success in further education and, for career-technical education 
programs, employment. Among the Washington colleges, however, the focus of 
instructional reforms was on improving student success in developmental education and 
introductory college-level math and English. Far less attention was paid to improving 
student readiness and success in other introductory college courses that serve as 
gatekeepers to programs in other fields such as social and behavioral sciences, 
biological and physical sciences, and business. On exception that should be noted was 
the Reading Apprenticeship work at WA-B, which involved faculty across disciplines, 
including those in career fields. 

Taking steps to strengthen programs of study in career fields would have the 
additional effect of broadening involvement by faculty in areas other than English, 
math, and developmental education in colleges where it is a separate department. 
Colleges also need to strengthen the mechanisms for helping students choose from 
among the many programs offered. The work by WA-C to restructure the intake and 
advising functions, to help students choose a program and earn college credits in it as 
soon as possible, offers a promising model. 

Support Efforts to Improve Instruction Using Program and Course 
Learning Outcomes 

One way to ensure that programs are well structured and aligned with outcome 
requirements is to support faculty to develop learning outcomes standards for courses and 
programs, assess students’ mastery of these outcomes, and use the results to improve 
instruction. This was one of the observations of the MDRC-CCRC report on the five-year 
progress of the Round 1 colleges (Rutschow et al., 2011). Compared with the Round 1 colleges, 
the Washington colleges as a group devoted more energy to improving instruction. As noted, 
however, most of these efforts were focused on English and mathematics. Promoting the 
development and use of learning outcomes would help increase involvement of faculty in 
improvement efforts across program areas. 

The survey results discussed in Chapter 4 did not show a substantial increase in any of 
the Washington colleges during their initial involvement in ATD in the extent to which faculty 
participated in discussions related to improving student outcomes. The survey also found little 
evidence of an increase in the use by faculty of student learning measures other than grades. The 
survey did show that the extent to which faculty used data for improvement was based on the 
policies and practices of their department; they were less influenced by the policies and 
practices of the central college administration. This finding is consistent with the fact that 
decision making on instruction in community colleges tends to be decentralized, with policies 
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that affect the classroom practice typically set at the department or division levels. So, to change 
instructional policy and practice, colleges will need to engage deans and department chairs (and 
not just faculty members) more extensively than did the Washington ATD colleges. 

Use a Common Set of Performance Measures to Focus Strategy 
Development and Promote Sharing 

As reported in Chapter 2, each of the Washington colleges has developed clear goals for 
student success with accompanying metrics, spurred in part by the new accreditation standards 
of the Northwest Commission on Colleges and Universities that emphasize the use of metrics 
and evidence in planning and improvement. While some of the colleges have borrowed from 
ATD performance measures, none has adopted them as a set. Moreover, to our knowledge, 
none of the colleges focused much attention on their performance according to the ATD 
metrics, instead focusing on their own measures. 

While it makes sense for colleges to adopt their own performance measures, from the 
standpoint of ATD it is problematic that there is not a common set of metrics that colleges 
across the initiative recognize and pay attention to. Unless colleges focus their reforms on 
particular metrics, it is unlikely that they will develop innovations that will improve their 
performance on those metrics. Also, without shared performance measures across colleges, it is 
difficult for the initiative and colleges participating in it to identify colleges that do improve 
their performance. 

Therefore, we recommend that Achieving the Dream encourage colleges across the 
initiative to adopt a common set of performance measures for use in developing and examining 
the effectiveness over time of their own strategies and in comparing colleges on particular 
measures for the purpose of identifying those that have improved their performance over time. 
One advantage of focusing on improving performance on institution-level aggregate measures 
is that colleges will be encouraged to make systemic changes involving multiple, coordinated 
changes in policy and practice aimed at benefiting large numbers of students instead of pursuing 
small-scale programmatic interventions that have little chance of improving outcomes on a 
substantial scale. 

We suggest that all the Washington State ATD colleges (both the original six and the 
ten new institutions) consider using the longitudinal cohort measures developed by the Research 
and Planning Commission for the Governance Institute for Student Success (GISS) meetings 
for trustees and presidents that were described in Chapter 2. The fact that these measures were 
developed by the Research and Planning Commission, which is comprised of institutional 
research staff from each college, might give them credibility across the system. The measures 
make use of the Student Achievement Initiative data reported by each college to the State Board 
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for Community and Technical Colleges and thus could be readily calculated and updated by 
college on a regular basis. 


Looking Ahead 

Making systemic changes in practice in the ways that most of the original Washington 
State ATD colleges already have already begun to do and in the other ways recommended here 
requires resources — for new and strengthened information systems, for professional 
development and training, and for coordination. The Washington ATD colleges have shown 
that even during a period of dramatic cuts in funding, they have been willing to make major 
changes in practice. They have done so by reallocating their scarce resources from practices that 
are not effective to those that are believed to be so. The strong commitment to student success 
of these colleges, combined with the interest of other colleges in the state in joining ATD and 
the willingness of College Spark and Lumina Foundation to support them, makes us optimistic 
that Washington’s community and technical colleges, along with the State Board for 
Community and Technical Colleges, will find ways in this difficult economic climate to make 
changes that are necessary to improve student outcomes. 
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Appendix A 

Framework for Measuring College Progress 
in Implementing the ATD Culture of Evidence Model 




The following is the framework for the interview protocol used in the field research. It 
is based on the one used in the baseline field research on the Washington ATD colleges and is 
essentially the same as the framework used in the second wave of field research on the first 
round of colleges to join the initiative. Using the same framework made it possible to assess the 
progress over time of the Washington colleges and to compare the extent to which they adopted 
the ATD culture of evidence model with that of the Round 1 colleges. 


Framework for Measuring College Progress 
in Implementing the ATD Culture of Evidence Model 



Related 


1 = little or none... 


ATD 

Principle 

Ratings of College Progress 

5 = high level/ 
a lot 


OVERALL ASSESSMENT OF COLLEGE S PROGRESS IN ACHIEVING THE DREAM 


Effect of Achieving the Dream 




College’s progress toward institutionalizing the ATD 
“culture of evidence” model of institutional 
effectiveness 




Extent to which college sought to/succeeded in 
improving equity of achievement by race/ethnicity and 
income groups 




Overall impact/value added of Achieving the Dream 



Sustainability of Achieving the Dream 




The college expects that its “culture of evidence” will 
continue after ATD is over 




The college expects to continue a majority of its 
strategies after ATD is over 



Overall Successes and Challenges with Achieving the Dream 




Level of successes with Achieving the Dream 




Level of challenges with Achieving the Dream 




Success/impact of Achieving the Dream strategies 



Usefulness of Achieving the Dream Inputs 




Coach and Data Facilitator 



Coach was helpful to college’s efforts to build a culture 
of evidence and improve student success 




Data facilitator was helpful to college’s efforts to build a 
culture of evidence and improve student success 



Framework continued on the next page 
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Related 

ATD 

Principle 

Ratings of College Progress 

1 = little or none... 
5 = high level/ 
a lot 



Meetings and Conferences 



Strategy Institute was helpful to the college’s efforts to 
build a culture of evidence and improve student 
success 




Other ATD events were helpful to the college’s efforts 
to build a culture of evidence and improve student 
success 




ATD Grant 



The role that the $400K grant from ATD played in the 
college’s efforts to implement ATD reforms 



IMPLEMENTATION OF ATD PRINCIPLES 


PRINCIPLE 1: COMMITTED LEADERSHIP 

VISION AND VALUES (1.1) AND COMMITMENT (1.2) 


PRESIDENT 



1.1a 

President’s support for a focus on improving student 
success 



1.1b 

President’s awareness of and commitment to 
improving achievement gaps for students of differing 
racial/ethnic and income groups 



1.2b 

President’s involvement in implementing Achieving the 
Dream goals at the college 




Effect of ATD on president’s commitment and vision for 
student success 



SENIOR LEADERSHIP (VPs, Deans) 



1.1a 

Senior leadership’s support for a focus on improving 
student success 



1.1b 

Senior leadership’s awareness of and commitment to 
improving achievement gaps for students of differing 
racial/ethnic and income groups 



1.2b 

Senior leadership’s involvement in implementing 
Achieving the Dream goals at the college 




Effect of ATD on senior leadership’s commitment and 
vision for student success 



BOARD 



1.2a 

Board’s awareness and support for improving student 
success 



1.2a 

Board’s exposure to reports on student outcomes 




Effect of ATD on Board’s awareness of/support for 
improving student success 



Framework continued on the next page 
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Related 


1 = little or none... 


ATD 

Principle 

Ratings of College Progress 

5 = high level/ 
a lot 


FACULTY AND STAFF 



1.2c 

Faculty leaders’ (e.g. department/division heads) 
support for improving student success 



1.2c, 3.1 

Faculty and staff involved in ATD support for improving 
student success 



1.2c, 3.1 

Faculty and staff NOT involved in ATD support for 
improving student success 



COMMUNICATION 



1.1c 

The college’s internal communication about improving 
student success 



1.1c 

The college’s external communication about improving 
student success 



PRINCIPLE 2: USE OF EVIDENCE TO IMPROVE, POLICIES, PROGRAMS, AND SERVICES 


IR AND IT CAPACITY (2. 1 ) 



2.1a 

IT’s ability to meet the demands for data and 
institutional research 



2.1b 

Integrity of policies/practices used to collect and 
securely store data 



2.1c 

IR’s ability to meet the demands for data and 
institutional research 



2. Id 

IR’s involvement in educating faculty and staff on 
research and data analysis 




Effect of ATD on increasing IR capacity 




Effect of ATD on college’s respect/need for IR 



PROCESS FOR IDENTIFYING ACHIEVEMENT GAPS (2.2) 



2.2a 

Longitudinal data on student outcomes collected and 
analyzed 



2.2b 

Data disaggregated for different student groups (e.g. 
age, race, gender, etc.) 



2.2c 

Surveys, focus groups, and other qualitative data 
collected and analyzed 




Effect of ATD on college’s overall data collection and 
analysis 




Effect of ATD on college’s overall awareness of 
achievement gaps 



Framework continued on the next page 
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Related 


1 = little or none... 


ATD 

Principle 

Ratings of College Progress 

5 = high level/ 
a lot 


PROCESS FOR FORMULATING AND EVALUATING SOLUTIONS (2.3) 



2.3a 

College involves larger campus community in 
reviewing data on student outcomes. 



2.3b 

College uses data on student outcomes to develop or 
refine policy and strategies. 




Effect of ATD on college’s use of data to improve 
programs and policies 



PRINCIPLE 3: BROAD ENGAGEMENT 


FACULTY AND STAFF (3. 1 ) 



3.1a 

Faculty’s involvement in examining program data and 
developing strategies 



3.1b 

Faculty’s use of research to assess programs and 
teaching 



3.1c 

Part-time/adjunct faculty’s involvement in efforts to 
improve student success 



3. Id 

Student services staffs’ use of research on effective 
practice to assess programs and strategies 



3.1e 

Collaboration between student services and 
faculty/instructional staff 




Effect of ATD on faculty’s involvement in efforts to 
improve student success 




Effect of ATD on PT/adjunct faculty involvement in 
efforts to improve student success 




Effect of ATD on student services involvement in 
efforts to improve student success 




Effect of ATD on the collaboration between student 
services and faculty 



STUDENTS AND EXTERNAL STAKEHOLDERS (3.2) 



3.2a 

Students’ involvement in efforts to improve student 
success 



3.2b 

Colleges’ attempts to bring in external stakeholders in 
efforts to improve student success 




Effect of ATD on students’ involvement in efforts to 
improve student success 




Effect of ATD on colleges’ attempts to involve external 
stakeholders in efforts to improve student success 



Table continued on the next page 
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Related 


1 = little or none... 


ATD 

Principle 

Ratings of College Progress 

5 = high level/ 
a lot 


PRINCIPLE 4: SYSTEMIC INSTITUTIONAL IMPROVEMENT 


INSTITUTIONAL MANAGEMENT (4. 1) 



4.1a 

College has a strategic planning process that uses 
data to inform measures and goals for student success 



4.1b 

College has a limited set of priorities to guide their 
work to improve student success 



4.1c 

College evaluates its programs and services to see if 
they improve student success 



4. Id 

College makes budget decisions based on evidence of 
program effectiveness 



4.1e 

College uses external grants to support improve broad 
efforts to improve student success 



4. If 

College has made efforts to scale up one or more of 
strategies 



4.1g 

College’s accreditation process is linked with its efforts 
to improve student success 




Effect of ATD on college’s strategic planning process 



ORGANIZATION (4.2) 



4.2a 

College has committees that oversee/monitor efforts to 
improve student success 




College has integrated the work of the core team with 
pre-existing or new strategic planning and/or 
committees to monitor student success 




College has integrated the work of the Data Team with 
strategic planning and/or committees to monitor 
student success 



4.2b 

Meetings, organizational units and workgroups 
regularly focus on student success 




Effect of ATD on college’s organizational structure 



PROFESSIONAL DEVELOPMENT (4.3) 



4.3a 

Professional development training reinforces efforts to 
improve student success 



4.3b 

Faculty/staff orientation activities promote a 
commitment to student success 



4.3c 

Faculty and staff receive training on using data and 
research 




Effect of ATD on college’s professional development 
offerings 
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Appendix B 

Survey Results on Data Use by Faculty and 
Administrators at the Washington ATD Colleges 




Achieving the Dream Colleges in Washington State 


Appendix Table B.l 


Changes in Faculty Views About the Use of Data on Students 
Between 2007-2008 and 2010-2011 


Survey Item 

WA-A 

WA-B 

WA-C 

WA-D 

WA-E 

WA-F 

Accessibility of data: 







The data in the college’s SIS are generally 
accurate and error free 

0 

0 

0 

+ 

0 

0 

The data 1 need are generally available in a 
user-friendly format 

+ 

0 

0 

+ 

0 

0 

The college’s IR staff is responsive to requests 
for information 

0 

+ 

0 

+ 

0 

0 

The college’s 1R department is adequately 
staffed for the college’s information and 
research needs 

0 

+ 

0 

+ 

0 

+ 

The reports and other information the college 
provides to administrators are typically clear 
and easy to follow 

0 

0 

0 

0 

0 

+ 

I am able to obtain the information I need in a 
timely fashion 

+ 

+ 

0 

+ 

0 

0 

Sources of data: 







Searches using the college’s student 
information system 

0 

0 

0 

0 

0 

0 

Data from the college web site or fact book 

0 

0 

0 

0 

0 

0 

Reports distributed by the college’s 
institutional research ( IR) office or other 
departments 

“ 

0 

0 

0 

0 

“ 

Requests to the IR or information technology 
(IT) staff 

0 

0 

0 

0 

0 

0 

My department’s database 

0 

0 

0 

0 

0 

0 

Student surveys or focus groups 

0 

- 

0 

0 

0 

0 

State databases or research reports 

+ 

0 

0 

0 

0 

0 

Other sources 

0 

- 

+ 

0 

0 

0 

I generally do not need information about 
groups of students 

0 

0 

+ 

0 

0 

0 


Table continued on the next page 
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Survey Item 

WA-A WA-B 

WA-C 

WA-D 

WA-E 

WA-F 

Participation in organized discussions about the following: 






Improving academic achievement or closing 
the achievement gaps 

- 

0 

0 

0 

0 

0 

Academic needs or performance of students of 
color 

0 

0 

0 

0 

0 

0 

Academic needs or performance of low- 
income students 

0 

0 

0 

0 

0 

0 

Reason for not using data: 







I am too busy with my teaching 
responsibilities 

0 

0 

- 

0 

0 

0 

It is not part of my responsibilities as a faculty 
member/administrator 

+ 

0 

0 

0 

0 

0 

1 do not have the quantitative and/or 
qualitative research skills to understand and 
use data and research 

0 

0 

0 

0 

0 

0 

1 do not ttust the data that are available 

0 

0 

0 

0 

0 

0 

The data that are available are not relevant to 
my role as a faculty member/administrator 

0 

0 

0 

0 

0 

- 


KEY: 

+ indicates a statistically significant positive change. 
- indicates a statistically significant negative change. 
0 indicates no change. 
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Achieving the Dream Colleges in Washington State 


Appendix Table B.2 


Changes in Administrator Views About the Use of Data on Students 
Between 2007-2008 and 2010-2011 


Survey Item 

WA-A 

WA-B 

WA-C 

WA-D 

WA-E 

WA-F 

Accessibility of data: 







The data in the college’s SIS are generally 
accurate and error free 

0 

+ 

0 

0 

0 

0 

The data 1 need are generally available in a 
user-friendly format 

0 

+ 

+ 

+ 

0 

0 

The college’s IR staff is responsive to 
requests for information 

0 

+ 

+ 

0 

0 

0 

The college’s IR department is adequately 
staffed for the college’s information and 
research needs 

0 

0 

+ 

0 

0 

0 

The reports and other information the 
college provides to administrators are 
typically clear and easy to follow 

0 

+ 

0 

0 

0 

0 

I am able to obtain the information I need in 
a timely fashion 

Sources of data: 

0 

+ 

+ 

0 

0 

0 

Searches using the college’s student 
information system 

* 

0 

0 

0 

0 

0 

Data from the college web site or fact book 

0 

0 

0 

+ 

0 

0 

Reports distributed by the college’s 
institutional research (IR) office or other 
departments 

0 

0 

' 

+ 

0 

0 

Requests to the IR or information 
technology (IT) staff 

0 

0 

0 

+ 

+ 

0 

My department’s database 

0 

0 

0 

0 

0 

0 

Student surveys or focus groups 

0 

0 

0 

+ 

0 

0 

State databases or research reports 

0 

0 

0 

0 

0 

0 

Other sources 

0 

0 

- 

0 

- 

0 

I generally do not need information about 
groups of students 

- 

- 

0 

0 

- 

- 


Table continued on the next page 
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Survey Item 

WA-A 

WA-B 

WA-C 

WA-D 

WA-E 

WA-F 

Participation in organized discussions about the following: 





Improving academic achievement/closing 
achievement gaps 

0 

0 

0 

+ 

0 

0 

Academic needs or performance of students 
of color 

0 

0 

0 

0 

0 

0 

Academic needs or performance of low- 
income students 

0 

0 

0 

0 

+ 

0 

Reason for not using data: 







I am too busy with my administrative 
responsibilities 

0 

0 

0 

- 

0 

0 

It is not part of my responsibilities as a 
faculty member/ administrator 

0 

0 

0 

0 

0 

0 

1 do not have the quantitative and/or 
qualitative research skills to understand and 
use data and research 

0 

0 



0 

0 

1 do not ttust the data that are available 

0 

0 

- 

0 

0 

0 

The data that are available are not relevant 
to my role as a faculty 
member/administrator 

0 

0 

0 

“ 

- 

0 


KEY: 

+ indicates a statistically significant positive change. 
- indicates a statistically significant negative change. 
0 indicates no change. 
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Achieving the Dream Colleges in Washington State 


Appendix Table B.3 


Faculty Members’ Reported Extent of Use of Data on Students in Making Teaching-Related Decisions 




WA-A 



WA-B 



WA-C 


Purpose for Data Use 

07 mean 

10 mean 

07-10 

07 mean 

10 mean 

07-10 

07 mean 

10 mean 

07-10 

Curriculum 

4.4 

4.8 

0.4* 

4.6 

4.7 

0.2 

4.4 

5.1 

Q 7** 

Teaching 

4.9 

5.3 

0.3 

5.1 

5.3 

0.2 

4.9 

5.4 

0.5* 

Advising 

4.7 

5.2 

0.5* 

4.9 

5.3 

0.4 

4.6 

5.2 

0.6* 

Identification of high failure 
rate courses 

4.6 

5.1 

0.5* 

4.9 

5.3 

0.4 

4.5 

5.2 

Q 7** 




WA-D 



WA-E 



WA-F 


Purpose for Data Use 

07 mean 

10 mean 

07-10 

07 mean 

10 mean 

07-10 

07 mean 

10 mean 

07-10 

Curriculum 

4.5 

5.0 

0.5* 

4.4 

5.0 

0.7 

4.4 

4.7 

0.3 

Teaching 

4.9 

5.4 

0.5** 

4.7 

4.9 

0.2 

4.5 

5.1 

0.6* 

Advising 

4.7 

5.7 

q 9*** 

4.6 

4.5 

-0.1 

4.5 

4.6 

0.1 

Identification of high failure 
rate courses 

4.7 

5.4 

Q 7** 

4.5 

4.5 

0.0 

4.2 

4.6 

0.4 


***p < .01. **p < .05. *p < .10. 
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Appendix Table B.4 


Faculty Perceptions of Their Departments’ Extent of Use of Data on Students in Making Decisions 




WA-A 



WA-B 



WA-C 


College Area 

07 mean 

10 mean 

07-10 

07 mean 

10 mean 

07-10 

07 mean 

10 mean 

07-10 

Curriculum 

4.2 

5.0 

Q g*** 

4.2 

4.9 

0.7 

5.0 

5.3 

0.3 

Teaching 

4.5 

5.1 

0.6** 

4.8 

4.9 

0.2 

5.2 

5.3 

0.0 

Tutoring 

4.4 

4.5 

0.2 

4.8 

4.8 

0.0 

5.1 

5.4 

0.4 

Program planning 

4.7 

5.1 

0.5* 

4.6 

4.8 

0.2 

5.6 

5.2 

-0.3 

Program evaluations 

4.4 

5.1 

q 7*** 

4.9 

5.2 

0.3 

5.4 

5.4 

-0.1 

Long-term planning 

4.3 

4.9 

0.6** 

4.6 

4.7 

0.1 

5.0 

5.2 

0.2 

Budgeting 

3.6 

4.4 

q 7** 

4.4 

4.9 

0.5 

4.7 

5.3 

0.5* 

Identification of high failure 
rate courses 

4.1 

4.7 

1.0** 

3.8 

4.2 

0.4 

4.8 

5.4 

0.6* 

Areas for improvement 

4.4 

4.8 

0.4 

4.5 

4.2 

-0.3 

5.0 

5.1 

0.1 




WA-D 



WA-E 



WA-F 


College Area 

07 mean 

10 mean 

07-10 

07 mean 

10 mean 

07-10 

07 mean 

10 mean 

07-10 

Curriculum 

4.9 

5.4 

0.5* 

4.2 

5.3 

\ i** 

4.4 

5.0 

0.6 

Teaching 

4.7 

5.3 

0.6** 

4.7 

4.9 

0.3 

4.3 

4.9 

0.5 

Tutoring 

4.3 

4.7 

0.3 

4.4 

4.6 

0.2 

4.4 

4.9 

0.5 

Program planning 

5.0 

5.3 

0.3 

4.8 

4.8 

0.0 

4.7 

4.9 

0.2 

Program evaluations 

4.7 

5.1 

0.3 

4.6 

4.8 

0.2 

5.0 

5.1 

0.1 

Long-term planning 

4.7 

5.1 

0.4 

4.5 

4.3 

-0.1 

4.7 

5.2 

0.5 

Budgeting 

4.0 

4.6 

0.6* 

3.7 

4.0 

0.3 

4.6 

5.1 

0.5 

Identification of high failure 
rate courses 

4.2 

5.5 


3.8 

4.9 

1.2** 

3.6 

4.5 

0.9** 

Areas for improvement 

4.4 

4.8 

0.4 

4.0 

4.4 

0.4 

4.5 

4.6 

0.1 


***p < .01. **p < .05. *p < .10. 
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Achieving the Dream Colleges in Washington State 

Appendix Table B.5 


Administrator Perceptions of Their Departments’ Extent of Use of Data on Students in Making Decisions 


College Area 


WA-A 



WA-B 



WA-C 


07 mean 

10 mean 

Mean diff. 

07 mean 

10 mean 

Mean diff. 

07 mean 

10 mean 

Mean diff. 

Curriculum 

4.7 

3.5 

- 1 . 2 ** 

4.8 

4.2 

- 0.6 

5.3 

3.5 


Program planning 

5.0 

3.6 

-1.4** 

5.8 

4.5 

-1.3* 

5.6 

3.9 


Program evaluations 

5.4 

3.6 

_ ] g*** 

6.3 

5.1 

- 1 . 2 * 

5.8 

4.1 


Long-term planning 

5.4 

5.5 

0.1 

5.5 

5.7 

0.2 

5.5 

4.5 

-0.9 

Budgeting 

5.4 

5.3 

- 0.1 

5.5 

5.7 

0.1 

5.6 

4.9 

-0.7 

Areas for improvement 

5.5 

5.6 

0.1 

5.0 

5.7 

0.7 

5.9 

4.6 

4 *** 


College Area 


WA-D 



WA-E 



WA-F 


07 mean 

10 mean 

Mean diff. 

07 mean 

10 mean 

Mean diff. 

07 mean 

10 mean 

Mean diff. 

Curriculum 

5.2 

3.2 

-2 0 *** 

4.1 

3.9 

- 0.2 

4.9 

3.9 

- 1 . 0 * 

Program planning 

5.3 

4.1 

- 1 . 2 * 

4.8 

4.5 

-0.4 

5.4 

3.8 

7 *** 

Program evaluations 

5.2 

3.6 

- 1 . 6 ** 

4.7 

5.6 

0.9 

5.0 

3.7 

-1.3** 

Long-term planning 

5.6 

5.7 

0.1 

4.9 

5.7 

0.8 

5.2 

5.5 

0.3 

Budgeting 

5.4 

5.8 

0.4 

5.0 

5.9 

0.9* 

5.5 

5.4 

- 0.1 

Areas for improvement 

5.7 

5.8 

0.1 

5.0 

5.7 

0.7 

5.3 

5.5 

0.2 


***p < .01. **p < .05. *p < .10. 
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Appendix C 

Additional Student Outcome Trends 
at the Washington ATD Colleges 




Achieving the Dream Colleges in Washington State 

Appendix Figure C.l 

Trends in Developmental English and Developmental Math Attempts 
Among All Students, 2003-2009: Washington ATD Colleges 

« Developmental English Attempts 
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SOURCE: MDRC calculations using the Achieving the Dream database maintained by JBI. Associates. 

NOTES: Calculations for these figures used institutional means for all available data for sample members in the fall 2003 through fall 2009 
cohorts at Achieving the Dream Round 3 Washington colleges. The vertical line shows the last cohort prior to the implementation of 
Achieving the Dream. WA-B and WA-D were excluded from the English developmental trend due to lack of data; WA-B was excluded from 
the math developmental trend due to a lack of data. 
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Achieving the Dream Colleges in Washington State 

Appendix Figure C.2 

Trends in Gatekeeper English and Gatekeeper Math Completions 
Among Students Referred to Developmental Courses, 2003-2009: 
Washington ATD Colleges 

— ♦— Gatekeeper English Completion (Referred) 



♦ Gatekeeper Math Completion (Referred) 



SOURCE: MDRC calculations using the Achieving the Dream database maintained by JBL Associates. 

NOTES: Calculations Cor these figures used institutional means lor all available data for sample members in the fall 2003 through fall 2009 
cohorts at Achieving the Dream Round 3 Washington colleges. The vertical line shows the last cohort prior to the implementation of 
Achieving the Dream. WA-B and WA-D were excluded from the English developmental trend due to lack of data; WA-B was excluded from 
the math developmental trend due to a lack of data. 


108 


Achieving the Dream Colleges in Washington State 
Appendix Figure C.3 

Trends in Credits Attempted and Completed Among All Students, 2003-2009: 
Two-Year Outcomes for Washington ATD Colleges 
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SOURCE: MDRC calculations using the Achieving the Dream database maintained by JBL Associates. 

NOTES: Calculations for these figures used institutional means for all available data for sample members in the fall 2003 through fall 2009 
cohorts at Achieving the Dream Round 3 Washington colleges. The vertical line shows the last cohort prior to the implementation of 
Achieving the Dream. WA-B was excluded from credit-related outcomes due to data concents. WA-C was excluded from 2009 credits 
attempted due to missing data. 
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Appendix Figure C.4 

Trends in Credit Milestones of 80 Percent Completion and 20 Credit Completion 
Among All Students, 2003-2009: Two-Year Outcomes for Washington ATD Colleges 
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SOURCE: MDRC calculations using the Achieving the Dream database maintained by JBL Associates. 

NOTES: Calculations for these figures used institutional means for all available data for sample members in the fall 2003 through fall 2009 
cohorts at Achieving the Dream Round 3 Washington colleges. The vertical line shows the last cohort prior to the implementation of 
Achieving the Dream. WA-B was excluded from credit-related outcomes due to data concerns. WA-C was excluded from 2009 completion of 
80% or more due to missing data. 
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Achieving the Dream Colleges in Washington State 

Appendix Figure C.5 

Trends in Ratios of Completed to Attempted Credits and Enrollment in the 
Second Year Among All Students, 2003-2009: 

Two-Year Outcomes for Washington ATD Colleges 

—♦—Ratio of Completed Credits to Attempted Credits 
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♦ Enrollment in Second Year 



SOURCE: MDRC calculations using the Achieving the Dream database maintained by JBL Associates. 

NOTES: Calculations lor these figures used institutional means for all available data for sample members in the fall 2003 through fall 2009 
cohorts at Achieving the Dream Round 3 Washington colleges. The vertical line shows the last cohort prior to the implementation of 
Achieving the Dream. WA-B was excluded from credit-related outcomes due to data concerns. WA-C was excluded from the 2009 credit 
ratio due to missing data. Enrollment in second year estimates for 2009 omit WA-B and WA-C due to missingness. 
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Achieving the Dream Colleges in Washington State 
Appendix Figure C.6 

Trends in Credential Completion by Type Among All Students, 2003-2009: 
Two-Year Outcomes for Washington ATD Colleges 

— ♦ — Completion of Certificate or Diploma 
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SOURCE: MDRC calculations using the Achieving the Dream database maintained by JBL Associates. 

NOTES: Calculations for these figures used institutional means for all available data for sample members in the fall 2003 through fall 2009 
cohorts at Achieving the Dream Round 3 Washington colleges. The vertical line shows the last cohort prior to the implementation of 
Achieving the Dream. The estimates for completion of credentials for the 2009 cohort at WA-B consist of only one year of data. 
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Appendix Figure C.7 

Trends in Pell Grant Receipt Among All Students, 2003-2009: 
Washington ATD Colleges 
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SOURCE: MDRC calculations using the Achieving the Dream database maintained by JBL Associates. 

NOTES: Calculations for these figures used institutional means for all available data for sample members in the fall 2003 through fall 2009 
cohorts at Achieving the Dream Round 3 Washington colleges. The vertical line shows the last cohort prior to the implementation of 
Achieving the Dream. 
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Appendix D 

Average Student Outcome Changes at 
Individual Washington ATD Colleges 




Achieving the Dream Colleges in Washington State 


Appendix Figure D.l 

Average Changes in Developmental English and Development Math Referrals 
Among All Students, 2003-2009: Washington ATD Colleges 


Post-Initiative Average Minus Pre-Initiative Average 
English Developmental Referral Rates, by Institution 



Post-Initiative Average Minus Pre-Initiative Average 
Math Developmental Referral Rates, by Institution 



Round 3 WA Colleges 


SOURCE: MDRC calculations using the Achieving the Dream database maintained by JBL Associates. 

NOTES: Calculations for these figures used institutional means for all available data for sample members in the fall 2003 through fall 2009 
cohorts at Achieving the Dream Round 3 Washington colleges. Percentage point change is calculated by subtracting the pre-intervention 
(2003 to 2006) average from the post-intervention (2007 to 2009) average. WA-B and WA-D were excluded from the English developmental 
trend due to lack of data; WA-B was excluded from the math developmental trend due to a lack of data. 
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Achieving the Dream Colleges in Washington State 
Appendix Figure D.2 

Average Changes in Developmental English and Development Math Completions 
Among Students Referred to Any Developmental English or Math Course, 2003-2009: 

Washington ATD Colleges 


Post-Initiative Average Minus Pre-Initiative Average 
English Developmental Completion Rates, by Institution 



Post-Initiative Average Minus Pre-Initiative Average 
Math Developmental Completion Rates, by Institution 



SOURCE: MDRC calculations using the Achieving the Dream database maintained by JBL Associates. 

NOTES: Calculations for these figures used institutional means for all available data for sample members in the fall 2003 through fall 2009 
cohorts at Achieving the Dream Round 3 Washington colleges. Percentage point change is calculated by subtracting the pre-intervention 
(2003 to 2006) average from the post-intervention (2007 to 2009) average. WA-B and WA-D were excluded from the English developmental 
trend due to lack of data; WA-B was excluded from the math developmental trend due to a lack of data. 
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Appendix Figure D.3 

Average Changes in Gatekeeper English and Gatekeeper Math Completions 
Among All Students, 2003-2009: Washington ATD Colleges 


Post-Initiative Average Minus Pre-Initiative Average 
English Gatekeeper Completion Rates, by Institution 



Post-Initiative Average Minus Pre-Initiative Average 
Math Gatekeeper Completion Rates, by Institution 



SOURCE: MDRC calculations using the Achieving the Dream database maintained by JBL Associates. 

NOTES: Calculations for these figures used institutional means for all available data for sample members in the fall 2003 through fall 2009 
cohorts at Achieving the Dream Round 3 Washington colleges. Percentage point change is calculated by subtracting the pre-intervention 
(2003 to 2006) average from the post-intervention (2007 to 2009) average. Gatekeeper course completion estimates for the 2009 cohort at 
WA-B consist of only one year of data. 
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Appendix Figure D.4 

Average Changes in Fall-to-Fall Persistence and Completion of Any Credential 
Among All Students, 2003-2009: Washington ATD Colleges 


Post-Initiative Average Minus Pre-Initiative Average 
Fall-to-Fall Persistence Rates, by Institution 
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SOURCE: MDRC calculations using the Achieving the Dream database maintained by JBL Associates. 

NOTES: Calculations for these figures used institutional means for all available data for sample members in the fall 2003 through fall 2009 
cohorts at Achieving the Dream Round 3 Washington colleges. Percentage point change is calculated by subtracting the pre-intervention 
(2003 to 2006) average from the post-intervention (2007 to 2009) average. The estimate for completion of any credential for the 2009 cohort 
at WA-B consists of only one year of data. The 2009 cohort at WA-C is not included in the fall-to-fall estimate due to missing data. 
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